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Diagnosing  Key  Drivers  of  Job  Impact  and  Business  Results 
Attributable  to  Training  at  the  Defense  Acquisition  University 

Nick  Bontis,  Chris  Hardy,  and  John  R.  Mattox 


The  Defense  Acquisition  University  (DAU)  is  an  integral  component  in  the 
career  of  every  Defense  Acquisition  Workforce  member,  from  the  time  they 
enroll  in  their  first  DAU  course  until  they  retire.  One  of  the  many  keys  to  DAU’s 
success  is  its  ability  to  measure  the  effectiveness  of  its  training  programs, 
monitor  performance,  and  improve  its  curriculum.  To  this  end,  the  authors 
conducted  a  data  mining  exercise  within  the  training  evaluation  data  to  deter¬ 
mine  the  key  drivers  of  its  success.  This  article  explains  the  methodological 
approach  used  (structural  equation  modeling)  as  well  as  the  results,  recom¬ 
mended  actions,  and  outcomes.  Within  the  DAU  learning  enterprise,  more  than 
326,000  training  events  were  evaluated  during  19  months  between  January 
1,  2008,  and  July  30,  2009.  Results  indicate  that  DAU’s  learning  enterprise 
positively  influences  job  impact  and  business  results. 

Motivating  the  Knowledge  Worker 

368  David  E.  Frick 


Commonly  accepted  economic  theory  suggests  that  workers  are  rational  actors 
and  make  decisions  that  will  maximize  expected  outcomes.  As  such,  managers 
should  be  able  to  influence  behaviors  to  meet  business  goals  by  manipulating 
the  expectations  of  outcomes.  Conversely,  social  science  practitioners  suggest 
that  workers  often  make  decisions  that  are  irrational.  Knowledge  workers  are 
a  growing  sector  of  the  workforce  and  are  the  backbone  for  entire  federal 
agencies.  The  acquisition  community  falls  within  this  category.  Identifying 
factors  that  influence  the  performance  of  knowledge  workers  may  be  critical 
to  maintaining  high  levels  of  organizational  performance.  This  research  focused 
on  identifying  the  factors  that  encourage  knowledge  workers  to  maintain  high 
levels  of  performance. 


Requirements  and  Cost  Stability: 

A  Case  Study  of  the  F/A-18  Hornet  Program 

388  CDR  Jay  D.  Bottelson,  USN 


Most  government  and  industry  leaders  involved  with  Department  of  Defense 
acquisition  programs  emphasize  the  importance  of  requirements  and  cost 
stability.  However,  despite  all  the  stated  support  for  program  element  stability 
and  acquisition  reform,  frequent  changes  are  experienced  in  acquisition 
programs  that  affect  the  final  end  product  in  terms  of  changes  to  unit  design, 
number  of  units  procured,  system  and  subsystem  capability,  as  well  as  affecting 
the  overall  cost  of  the  program.  This  study  analyzes  the  U.S.  Navy’s  F/A-18A 
model  to  identify  requirements  changes;  discern  the  reasons  for  change  and 
the  impact  the  resultant  change  made  on  the  program  (funding,  schedule, 
capacity,  etc.);  and  develop  recommendations  for  limiting  requirements  creep, 
instability,  and  cost  growth  in  future  programs. 
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Better  Schedule  Performance  Assessments  Derived  from  Integrated 
Master  Plan-Referenced  Schedule  Metrics 

David  C.  Bachman 


The  integrated  master  plan  (IMP)  provides  a  better  structure  than  either  the 
work  breakdown  structure  (WBS)  or  organizational  breakdown  structure 
for  measuring  actual  integrated  master  schedule  (IMS)  progress.  The  author 
posits  that  improved  understanding  of  schedule  performance  and  better 
identification  of  program  risks  result  when  an  IMP  structure  is  evaluated  in 
addition  to  the  earned  value  management-mandated  IMS  WBS  structure.  The 
article  examines  how  the  “Hit-Miss”  index,  baseline  execution  index,  and  critical 
path  length  index  (CPLI)  were  used  to  evaluate  the  life-cycle  performance 
of  a  12-month,  900-task  IMP  program  event.  CPLI,  the  author  concludes,  is 
subject  to  interpretation  and  must  be  evaluated  against  four  caveats:  duration 
remaining,  total  float  including  schedule  margin,  schedule  compression,  and 
schedule  avoidance. 
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Reimagining  Workload  Task  Analysis: 
Applications  to  Training  System  Design 

Dennis  Duke,  Dana  E.  Sims,  and  James  Pharmer 


Today’s  warfighter  performs  more  complex,  cognitively  demanding  tasks  than 
ever  before.  Despite  the  need  for  more  extensive  training  to  perform  these 
tasks,  acquisition  professionals  are  often  tasked  to  reduce  training  budgets 
and  identify  optimal  tradeoffs.  Tools  are  available  to  help  them  make  these 
decisions  that  provide  empirical  evidence  of  how  performance  and  mission 
requirements  will  be  affected  by  design  decisions.  This  article  offers  insights 
into  the  utility  of  implementing  a  Workload  Task  Analysis  (WLTA)  early  in 
weapon  systems  acquisition  for  the  purpose  of  focusing  on  training  system 
decisions,  and  provides  a  description  of  where  WLTA  occurs  within  the  top- 
down  functional  analysis  process.  It  concludes  with  several  examples  of  how 
the  WLTA  results  can  be  used  to  guide  training  development. 
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From  the  Chairman  and  Executive  Editor 


“From  Workforce  to  Warfighter” 
describes  the  end-to-end  process  of 
developing  our  nation’s  defense  capa¬ 
bilities.  It  begins  long  before  a  user 
requirement  is  ever  written— in  the  hir¬ 
ing,  training,  and  retention  of  our  Defense 
Acquisition  Workforce.  It  continues  long 
after  a  system  or  service  is  produced— in 
the  hands  and  mind  of  the  warfighter  downrange. 

The  first  two  articles  deal  with  the  Defense  Acquisition 
Workforce.  Nick  Bontis,  Chris  Hardy,  and  John  Mattox  demon¬ 
strate  how  effective  training  of  this  workforce  contributes  to 
improvements  in  job  performance  and  business  results.  David 
Frick,  in  his  article,  argues  that  in  order  to  keep  this  workforce 
(primarily  composed  of  “knowledge  workers”)  motivated, 
traditional  approaches  such  as  pay-for-performance  are 
increasingly  anachronistic  and  may  need  to  be  discarded, 
while  more  nontraditional  approaches  need  to  be  examined. 

The  next  two  articles  take  to  task  long-held  assump¬ 
tions  about  the  defense  acquisition  process.  Jay  Bottelson 
examines  the  Navy’s  F/A-18  Hornet  program  in  terms  of 
requirements  creep  and  cost  growth,  and  finds  that  the  prob¬ 
lems  occurred  in  unexpected  ways.  David  Bachman  argues 
that  using  integrated  master  plans  provides  more  meaningful 
metrics  for  measuring  near-term  schedule  performance  than 
the  traditional  management  tools  of  work  and  organizational 
breakdown  structures.  Finally,  Dennis  Duke,  Dana  Sims,  and 
James  Pharmer  take  us  downrange,  discussing  how  workload 
task  analysis  can  help  the  decision-making  process  for  devel¬ 
oping  warfighter  training  systems. 

The  latest  addition  to  the  Defense  Acquisition  Profes¬ 
sional’s  Reading  List  is  Neil  Sheehan’s  comprehensive  look 
at  the  cold  war  development  of  the  Intercontinental  Ballistic 
Missile,  A  Fiery  Peace  in  a  Cold  War:  Bernard  Schriever  and 
the  Ultimate  Weapon,  reviewed  here  by  James  Dobbins. 

Please  take  note  of  two  Calls  for  Papers: 


1.  “Doing  More  without  More:  Government  and  Industry 
Imperatives  for  Achieving  Acquisition  Efficiencies,” 
Defense  Acquisition  University  Alumni  Association 
2012  Hirsch  Research  Paper  Competition,  which  will 


take  place  at  the  DAU  main  campus,  Fort  Belvoir, 
Virginia,  in  April  2012.  Please  submit  papers  by 
November  1,  2011. 

2.  “The  Limits  of  Competition  in  Defense  Acquisition” 
Research  Symposium,  which  will  also  take  place  at  the 
DAU  main  campus,  Fort  Belvoir,  Virginia,  September 
18-19,  2012.  Please  submit  abstracts  by  November 
30,  2011. 

In  this  issue,  we  thank  the  men  and  women  from  across 
our  defense  acquisition  community,  who  have  taken  their 
time  and  effort  to  review  and  critique  the  articles  that  have 
graced  our  pages  in  the  past  year.  Their  dedication  helps  us 
ensure  that  our  readers  are  provided  with  the  highest  caliber 
research  and  analysis  that  can  help  inform  their  acquisition 
decision  making. 

We  also  take  this  opportunity  to  thank  Ed  Boyd,  the  Direc¬ 
tor  of  DAU  Visual  Arts  and  Press,  who  is  retiring  from  public 
service  at  the  end  of  this  year.  Since  2003,  he  has  been  the 
steady  hand  at  the  helm  of  DAU’s  publications  program,  help¬ 
ing  us  publicize  and  promote  defense  acquisition  from  the 
Defense  Acquisition  Workforce  all  the  way  to  the  warfighter. 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 


From  the  Chairman  and  Executive  Editor 


FAREWELL 


Eduard  Boyd  is  the  Director  of  Visual  Arts 
and  Press  at  the  Defense  Acquisition  University 
(DAU).  He  has  been  with  the  university  inter¬ 
mittently  since  1977  and  steadily  since  1994.  Mr. 

Boyd  served  in  the  U.S.  Army  for  22  years,  the 
majority  of  which  he  spent  gaining  experience 
as  an  illustrator.  He  has  contributed  illustration 
support  to  both  the  Defense  AT&L  magazine 
and  the  Defense  Acquisition  Research  Journal  since  its  inception  in 
1994.  Mr.  Boyd  originally  came  to  DAU  as  an  illustrator  while  he  was  still 
in  the  military.  Upon  retirement  from  the  U.S.  Army,  he  joined  DAU  as  a 
visual  information  specialist  in  the  federal  civilian  service.  Mr.  Boyd  was 
on  the  cusp  of  the  generation  actively  involved  in  transitioning  illustra¬ 
tion  from  a  manual  art  to  advanced  computer  multimedia,  distributed 
in  a  myriad  of  delivery  platforms.  He  embraced  this  evolving  culture 
and  became  an  expert  on  several  software  platforms,  i.e.,  Illustrator, 
InDesign,  which  enhanced  the  publications  program  immeasurably 
and  created  a  coherent  branding  for  the  university.  His  expertise  in 
planning  and  co-designing  DAU’s  convention  and  seminar  exhibits  dur¬ 
ing  the  past  20  years  won  him  praise  and  accolades  from  the  highest 
levels  of  the  Department  of  Defense,  including  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics.  Because  of  his 
in-depth  knowledge  of  the  organization  and  proven  managerial  skills, 
Mr.  Boyd  was  promoted  in  2005  to  Director  of  Visual  Arts  and  Press 
at  DAU.  He  is  the  recipient  of  several  awards  during  his  career  with  the 
university,  including  numerous  outstanding  and  superior  performance 
awards,  on-the-spot  cash  awards,  and  the  Commander’s  Award  for 
Civilian  Service.  Mr.  Boyd  is  also  a  two-time  recipient  of  the  Enlisted 
Person  of  The  Year  award,  as  well  as  a  four-time  recipient  of  the  En¬ 
listed  Person  of  the  Quarter  award. 
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DAUAA  2012 
RESEARCH  PAPER 
COMPETITION 


DOING  MORE 
WITHOUT  MORE 

Government  and  Industry  Imperatives 
for  Achieving  Acquisition  Efficiencies 


CALL  FOR  PAPERS 

Prospective  authors  are  invited  to  submit  papers  on  the  following  topics: 

•  Affordability  and  Control  of  Cost 

•  Improving  the  Acquisition  of  Services 

•  Competition  in  Defense  Acquisition 

•  Productivity  and  Innovation  in  Defense- Related  Industries 

•  Acquisition  Process 


GROUND  RULES 

•The  competition  is  open  to  anyone  interested 
in  the  DoD  acquisition  system  and  is  not  lim¬ 
ited  to  government  or  contractor  personnel. 

•Employees  of  the  federal  government  (includ¬ 
ing  military  personnel)  are  encouraged  to 
compete  and  are  eligible  for  cash  awards  unless 
the  paper  was  researched  or  written  as  part 
of  the  employee’s  official  duties  or  was  done 
on  government  time.  If  the  research  effort 
is  performed  as  part  of  official  duties  or  on 
government  time,  the  employee  is  eligible  for  a 
non-cash  prize,  i.e.,  certificate  and  donation  of 
cash  prize  to  a  Combined  Federal  Campaign-reg¬ 
istered  charity  of  winner's  choice 


•First  prize  is  $1,000.  Second  prize  is  $500. 

•The  format  of  the  paper  must  be  in  accordance 
with  guidelines  for  articles  submitted  for  the 
Defense  Acquisition  Research  Journal. 

•Research  papers  are  due  November  I,  2011. 

•Papers  are  to  be  submitted  to  the  DAU  Direc¬ 
tor  of  Research:  research@dau.mil. 

•  Papers  will  be  evaluated  by  a  panel  selected  by 
the  DAUAA  Board  of  Directors  and  the  DAU 
Director  of  Research. 

•Winners  will  be  announced  and  papers  will  be 
presented  at  the  DAU  Acquisition  Community 
Symposium  in  April  2012. 
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DIAGNOSING 
KEY  DRIVERS  OF 
JOB  IMPACT  AND 
BUSINESS  RESULTS 
ATTRIBUTABLE  TO 
TRAINING  AT  THE 
DEFENSE  ACQUISITION 
UNIVERSITY 

f  Nick  Bonds,  Chris  Hardy,  and  John  R.  Mattox 

The  Defense  Acquisition  University  (DAU)  is  an  integral  compo¬ 
nent  in  the  career  of  every  Defense  Acquisition  Workforce 
member,  from  the  time  they  enroll  in  their  first  DAU  course 
until  they  retire.  One  of  the  many  keys  to  DAU’s  success  is  its 
ability  to  measure  the  effectiveness  of  its  training  programs, 
monitor  performance,  and  improve  its  curriculum.  To  this  end, 
the  authors  conducted  a  data  mining  exercise  within  the  training 
evaluation  data  to  determine  the  key  drivers  of  its  success.  This 
article  explains  the  methodological  approach  used  (structural 
equation  modeling)  as  well  as  the  results,  recommended 
actions,  and  outcomes.  Within  the  DAU  learning  enterprise, 
more  than  326,000  training  events  were  evaluated  during  19 
months  between  January  1,  2008,  and  July  30,  2009.  Results 
indicate  that  DAU’s  learning  enterprise  positively  influences 
job  impact  and  business  results. 
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The  Defense  Acquisition  University  (DAU)  is  critical  to  ensur¬ 
ing  the  Defense  Acquisition  Workforce  is  well  trained  to  meet  our 
nation’s  needs.  As  such,  DAU  is  fully  integrated  in  the  careers  of  its 
workforce  from  the  time  they  enroll  in  their  first  DAU  course  until 
they  retire.  One  of  the  many  keys  to  DAU’s  success  is  its  ability  to 
measure  the  effectiveness  of  its  training  programs,  monitor  perfor¬ 
mance,  and  improve  its  curriculum  (DAU,  2010).  To  this  end,  DAU 
conducted  a  data  mining  exercise  with  its  training  evaluation  data 
to  determine  the  key  drivers  of  its  success. 

DAU  measures  and  monitors  its  own  performance  by  administer¬ 
ing  a  state-of-the-art,  end-of  course  survey  instrument,  which  is  a 
Web-based  learning  evaluation  system  with  an  extensive  database 
of  performance  benchmarks  collected  from  student  survey  data. 
DAU  evaluates  customer  satisfaction  based  on  the  4-level  Kirkpat¬ 
rick  training  assessment  model  and  uses  survey  items  on  a  7-point 
Likert-type  scale  (Kirkpatrick,  1998).  Students  are  provided  a  link  to 
the  survey  at  the  end  of  each  course,  which  includes  questions  on 
course  content,  quality  of  faculty,  and  job  applicability.  Ratings  are 
reviewed  regularly,  and  improvements  are  made  in  DAU’s  learning 
products  and  services  based  on  these  evaluations. 

This  study  focuses  on  the  evaluation  of  key  drivers  for  success¬ 
ful  training  events.  An  advanced  statistical  approach  called  causal 
modeling  (i.e.,  structural  equation  modeling)  was  used  to  determine 
relationships  among  latent  constructs  and  isolate  likely  paths  of 
causation.  The  main  objectives  of  this  research  study  were  fourfold. 

1.  Evaluate  the  survey  instrument  DAU  uses  and  determine 
whether  or  not  it  is  a  valuable  instrument  to  provide  infor¬ 
mation  for  decision  making; 

2.  Assess  the  relationship  between  job  performance  and  impact 
as  perceived  by  participants  attributed  to  DAU  training; 

3.  Access  the  antecedents  and  outcomes  of  learning  and  pro¬ 
vide  a  benchmarking  analysis  of  DAU  scores  versus  other 
organizations;  and 

4.  Provide  recommendations  for  isolated  points  of  intervention, 
which  will  yield  the  largest  improvements  for  the  learning 
enterprise. 


Development  of  Hypotheses 

Although  the  investment  in  learning  by  various  organizations  is 
far  from  consistent  across  industries  (or  even  across  departments 
within  the  same  organization),  few  would  argue  as  to  its  importance. 
The  resultant  training  and  development  budget  isolates  this  invest¬ 
ment  and  is  often  referred  to  when  senior  leaders  are  questioned 
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as  to  their  commitment  for  human  capital  development.  The  two 
primary  expenditures  related  to  training  are  instruction  and  materi¬ 
als.  High  levels  of  instructor  quality  are  synonymous  with  effective 
learning.  This  positivist  relationship  is  at  the  foundation  of  why 
instructors  are  continually  evaluated.  The  performance  of  a  teacher 
as  expressed  by  a  student  after  a  course  is  complete  is  a  universally 
adopted  founding  tradition  of  every  educational  institution.  For  this 
reason,  a  common  expectation  is  the  existence  of  a  positive  rela¬ 
tionship  between  instructor  effectiveness  and  individual  learning. 

Hypothesis  No.  1 

There  is  a  positive  relationship  between  instructor  effectiveness 
and  individual  learning.  In  addition  to  the  quality  of  instruction, 
courseware  quality  is  also  an  important  antecedent  to  learning. 
Whether  the  materials  are  physical  in  the  form  of  books  and  notes, 
or  online,  students  grasp  difficult  concepts  by  reading  them  over 
and  over  again.  While  the  instructor  may  reinforce  the  importance 
of  the  text,  the  explicit  documents  act  as  a  permanent  record  of  the 
content  that  a  student  is  expected  to  master.  As  such,  four  addi¬ 
tional  important  hypotheses  regarding  business  results  and  impact 
were  also  tested. 

Hypothesis  No.  2 

There  is  a  positive  relationship  between  courseware  quality 
and  individual  learning.  Given  the  assumption  that  an  instructor  is 
competent  and  that  course  materials  are  adequate,  students  often 
have  pre-conceived  notions  with  regards  to  the  value  of  a  course 
before  it  has  been  completed.  To  be  accurate,  the  perception  of  a 
worthwhile  investment  is  more  than  just  the  cost  of  the  registration 
fee.  In  most  cases,  the  opportunity  cost  of  time  while  sitting  through 
a  course  (and  therefore,  not  doing  the  job)  is  often  more  valuable 
to  the  learner.  Only  when  both  these  perceptions  (i.e.,  the  cost  of 
registration  and  the  opportunity  cost  of  time)  are  deemed  to  be  fair 
and  adequate,  can  a  student  realize  a  satisfactory  learning  experi¬ 
ence.  Therefore,  the  perception  of  a  worthwhile  investment  is  also 
expected  to  have  a  positive  relationship  with  individual  learning. 

Hypothesis  No.  3 

There  is  a  positive  relationship  between  worthwhile  investment 
and  individual  learning.  Sustainable  high  levels  of  organizational 
performance  can  be  attributable  in  large  part  to  a  superior  learning 
enterprise  that  transforms  human  capital  development  into  action¬ 
able  job  impact  and  business  results  (Bontis  &  Fitz-enz,  2002).  It 
follows  then  that  an  expected  positive  relationship  between  individ¬ 
ual  learning  and  job  impact  and  business  results  should  be  realized. 


October  2011 


351 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


352 


The  issue  here  is  one  of  temporal  lag.  How  do  learners  know  for 
sure  if  a  course  will  impact  their  job  later?  One  way  to  deal  with  this 
limitation  is  to  assess  the  outcomes  longitudinally.  In  other  words, 
provide  learners  with  an  opportunity  to  predict  an  outcome  imme¬ 
diately  after  the  course  is  complete,  and  then  again  sometime  into 
the  future,  a  retrospective  analysis.  As  such,  the  following  important 
hypotheses  regarding  business  results  and  impact  were  also  tested: 

Hypothesis  No.  4 

There  is  a  positive  relationship  between  individual  learning  and 
future  job  impact. 

Hypothesis  No.  5 

There  is  a  positive  relationship  between  individual  learning  and 
future  business  results. 

Hypothesis  No.  6 

There  is  a  positive  relationship  between  individual  learning  and 
actual  job  impact. 

Hypothesis  No.  7 

There  is  a  positive  relationship  between  individual  learning  and 
actual  business  results. 


Method 

When  analyzing  large  sets  of  data,  a  variety  of  statistical  tech¬ 
niques  are  available.  One  common  approach  used  by  researchers 
is  null  hypothesis  testing  with  experimental  and  quasi-experimental 
designs  (Shadish,  Cook,  &  Campbell,  2002).  An  alternative  approach 
is  data  modeling  (Rodgers,  2010).  Structural  equation  modeling 
(SEM)  is  one  such  approach  and  is  useful  for  many  reasons.  Fore¬ 
most  is  its  ability  to  test  multiple  hypotheses  simultaneously  and 
produce  a  visual  model  of  the  causal  relationships  within  a  data  set. 
While  the  benefits  of  SEM  are  plentiful,  a  significant  level  of  inter¬ 
pretation  of  these  models  is  necessary.  In  its  simplest  form,  SEM  is 
an  advanced  statistical  technique  that  computes  the  mathematical 
relationships  among  multiple  variables  simultaneously  in  order  to 
describe  a  chain  of  causation. 

The  measurement  and  structural  models  were  estimated  by 
using  Partial  Least  Squares  (PLS).  PLS  is  a  second  generation  SEM 
technique  that  has  received  positive  recognition  in  the  scientific 
community  (Chin,  1998;  Gefen,  Straub,  &  Boudreau,  2000).  PLS 
was  developed  by  Wold  (1975),  and  it  focuses  on  maximizing  the 
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variance  of  the  dependent  variables  explained  by  the  independent 
ones.  It  is  a  rigorous  SEM  technique  that  requires  only  minimal 
assumptions  about  the  distribution  of  the  data.  PLS  has  five  main 
advantages  over  other  covariance  methods  (e.g.,  LISREL,1  AMOS,2 
etc.)  for  this  research  study:  (a)  it  does  not  assume  normally  dis¬ 
tributed  raw  data;  (b)  the  presence  of  multicollinearity  in  the  data 
is  handled  well;  (c)  it  is  better  suited  to  explain  complex  exploratory 
relationships;  (d)  it  allows  variable  weights  to  scale  for  indicators; 
and  (e)  it  allows  the  use  of  noninterval  scales.  The  raw  data  set  that 
was  to  be  analyzed  fit  well  with  the  corresponding  advantages  of 
PLS.  Furthermore,  PLS  is  often  used  in  exploratory  research  with 
the  ultimate  goal  to  maximize  the  explanatory  power  of  the  resul¬ 
tant  model.  PLS  also  benefits  from  considering  all  path  coefficients 
simultaneously,  allowing  analysis  of  direct,  indirect,  and  spurious 
relationships  and  the  estimation  of  multiple  individual  item  load¬ 
ings  in  the  context  of  a  theoretically  specified  model  rather  than 
in  isolation  (enabling  researchers  to  avoid  biased  and  inconsistent 
parameter  estimates  for  equations). 


Results 

Whereas  in  traditional  path  analysis  the  calculation  of  reliability 
and  validity  statistics  is  independent  of  the  model  being  tested,  PLS 
generates  a  variety  of  reliability  and  validity  statistics  calculated  in 
the  context  of  the  theoretical  model  under  investigation.  To  vali¬ 
date  the  measurement  model,  the  authors  executed  the  following 
series  of  steps.  First,  construct  reliability  was  assured  by  calculating 
Cronbach’s  alpha  values  for  each  construct.  Cronbach’s  alpha  is 
often  used  to  confirm  that  respondents  interpreted  the  meaning  of 
survey  items  accurately,  and  would  continue  to  do  so  in  the  future. 
In  other  words,  survey  items  were  understood  by  the  respondents 
and  tended  to  hang  together  in  a  cluster.  A  Cronbach’s  alpha  value 
exceeding  0.70  is  considered  the  minimum  threshold  (Cronbach, 
1951).  Table  la  outlines  that  all  items  and  their  corresponding  con¬ 
structs  used  in  this  study  exceeded  0.70  (in  fact,  most  exceeded 
0.90). 

Loading  values  (lamdas)  were  used  to  measure  the  validity  of 
items.  This  test  examines  whether  or  not  the  survey  items  used  to 
evaluate  the  effectiveness  of  the  instructor  (four  items  in  this  case) 
load  on  to  an  overall  construct  about  the  instructor.  The  opposite 
would  be  if  a  survey  item  used  to  measure  the  quality  of  instructors 
actually  did  a  better  job  of  measuring  courseware  quality.  Again,  a 
measure  of  0.70  or  higher  is  desired,  and  this  minimum  threshold 
was  exceeded  in  all  cases  (Nunnally,  1978).  The  results  of  Table  1b 
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TABLE  1A.  MEASURES  OF  RELIABILITY  AND  VALIDITY  FOR 
EXOGENOUS  CONSTRUCTS 


ID 

Metric  (Base,  DAU) 

Lamda 

Instructor  Effectiveness  (Alpha  =  0.946,  0.931) 

Base 

DAU 

1058P 

The  instructor  was  knowledgeable  about 
the  subject. 

0.895 

0.910 

1059P 

The  instructor  was  prepared  and  organized 

for  the  class. 

0.885 

0.923 

1269P 

The  instructor  was  responsive  to 
participants'  needs  and  questions. 

0.856 

0.913 

1270P 

The  instructor's  energy  and  enthusiasm 
kept  the  participants  actively  engaged. 

0.851 

0.906 

Courseware  Quality  (Alpha  =  0.899,  0.798) 

1065P 

The  examples  presented  helped  me 

understand  the  content. 

0.886 

0.805 

2726P 

The  scope  of  the  material  was  appropriate 
to  my  needs. 

0.891 

0.806 

2730P 

The  participant  materials  (manual, 
handouts,  etc.)  will  be  useful  on  the  job. 

0.845 

0.800 

2924P 

The  material  was  organized  logically. 

0.892 

0.779 

Worthwhile  Investment  (Alpha  =  0.958,  0.971) 

2743P 

This  training  was  a  worthwhile  investment 
in  my  career  development. 

0.980 

0.986 

2744P 

This  training  was  a  worthwhile  investment 
for  my  employer. 

0.979 

0.986 

Individual  Learning  (Alpha  =  1.000, 1.000) 

919P 

1  learned  new  knowledge  and  skills  from 
this  training. 

1.000 

1.000 

illustrate  that  both  the  survey  instrument  and  DAU  models  used 
valid  and  reliable  measurement  instruments.  In  essence,  the  psy¬ 
chometric  evaluation  of  the  scales  used  in  this  study  was  successful 
and  therefore  adequate  for  model  interpretation. 

Survey  Instrument  vs.  DAU  Model  Interpretation 

In  2009,  Dr.  Nick  Bontis  created  a  predictive  learning  analytics 
model  that  describes  the  relationship  between  training  and  business 
performance  (Bontis  &  KnowledgeAdvisors,  2009).  The  analysis 
proved  successful  and  the  resulting  model  is  depicted  in  Figure  1.  As 
one  might  expect,  the  model  is  complex.  However,  the  model  is  rela¬ 
tively  easy  to  decipher  with  one  key  piece  of  information.  The  chain 
of  causation  lies  along  the  pathways  with  the  highest  beta  values. 
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TABLE  IB.  MEASURES  OF  RELIABILITY  AND  VALIDITY  FOR 
ENDOGENOUS  CONSTRUCTS 


ID 

Metric 

Lamda 

Perceived  Future  Job  Impact  (Alpha  =  0.832,  0.833) 

Base 

DAU 

712P 

1  will  be  able  to  apply  the  knowledge  and  skills 
learned  in  this  class  to  my  job. 

0.726 

0.749 

1279P 

How  critical  is  applying  the  content  of  this 
training  to  your  job  success?  0%-100% 

0.786 

0.925 

1423P 

What  percent  of  your  total  work  time  requires 
the  knowledge  or  skills  presented  in  this 
training?  0%-100% 

0.840 

0.898 

2788P 

What  percent  of  new  knowledge  and  skills 
learned  from  this  training  do  you  estimate  you 
will  directly  apply  to  your  job?  0%-100% 

0.873 

0.907 

Perceived  Future  Business  Results  (Alpha  =  0.775,  0.805) 

2740  P 

Estimate  how  much  you  expect  your  job 
performance  related  to  the  course  subject 
matter  to  improve  in  the  next  12  months. 

0%-100% 

0.909 

0.928 

2741 P 

Based  on  your  response  to  the  prior  question, 
how  much  of  the  improvement  will  be  a  direct 
result  of  this  training,  as  opposed  to  other 

factors?  0%-100% 

0.898 

0.901 

Job  Impact  in  60  days  (Alpha  =  0.892,  0.929) 

1737F 

What  percent  of  your  total  work  time  have  you 
spent  on  tasks  that  require  the  knowledge/ 
skills  presented  in  the  training?  Check  only  one. 

0%-100% 

0.889 

0.932 

1738F 

On  a  scale  of  0%  (not  at  all)  to  100%  (extremely 
critical),  how  critical  was  applying  the  content  of 
the  training  to  your  job  success?  Check  only  one. 

0.916 

0.934 

2818F 

What  percent  of  new  knowledge  and  skills 
learned  from  this  training  did  you  directly  apply 
to  your  job?  Check  only  one.  0%-100% 

0.912 

0.942 

Business  Results  in  60  days  (Alpha  =  0.708,  0.814) 

2751 F 

Given  all  factors,  including  this  training,  estimate 
how  much  your  job  performance  related  to  the 
course  subject  matter  has  improved  since  the 
training.  0%-100% 

0.906 

0.931 

2502F 

Based  on  your  response  to  the  prior  question, 
estimate  how  much  of  the  improvement  was  a 
direct  result  of  this  training.  0%-100% 

0.845 

0.905 
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Note.  Adapted  from  The  Predictive  Learning  impact  Model  by  N.  Bontis  and 
KnowledgeAdvisors,  Inc.  Copyright  2009  by  KnowledgeAdvisors,  Inc. 


Before  examining  the  model,  a  description  of  the  data  source  is 
essential.  Data  used  in  the  analysis  were  extracted  from  the  survey 
instrument  system.  Using  Web-hosted  surveys,  learners  provided 
feedback  about  their  training  experience  immediately  after  training 
and  60  days  after  training.  The  immediate  survey  asked  learners 
to  rate  the  quality  of  their  training  experience  as  well  as  predict 
whether  training  would  improve  their  job  performance  and,  in  turn, 
contribute  to  business  results.  In  the  model  shown  in  Figure  1,  the 
immediate  survey  results  contribute  to  the  following  factors  about 
training:  Instructor  Effectiveness,  Courseware  Quality,  Worthwhile 
Investment,  and  Individual  Learning. 

The  immediate  responses  also  contributed  estimates  of  future 
job  performance  and  estimated  impacts  on  business  results:  Per¬ 
ceived  Future  Job  Impact  and  Perceived  Future  Business  Results. 
The  far  right  side  of  the  model  has  two  factors,  Job  Impact  in  60 
Days  and  Business  Results  in  60  Days,  which  represent  retrospective 
input  gathered  from  learners  after  they  have  had  60  days  to  apply 
their  learning  on  the  job.  More  than  a  million  data  points  were  used 
in  the  survey  instrument  base  model  with  learners  assessed  from 
many  well-known,  globally  recognized  companies  (e.g.,  Microsoft, 
HSBC,  Caterpillar,  and  BAE). 
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The  left  side  of  the  model  represents  the  three  most  important 
antecedent  aspects  of  training:  Worthwhile  Investment,  Courseware 
Quality,  and  the  learner’s  perspective  about  Instructor  Effectiveness. 
But  how  do  these  three  factors  contribute  to  individual  learn¬ 
ing?  Effective  instructors  contribute  by  developing  high-quality 
courseware  (B  =  0.556)  (e.g.,  materials,  delivery  format,  learning 
environment,  etc.).  Both  contribute  to  the  learner’s  perception  that 
training  was  a  worthwhile  investment  (B  =  0.625).  While  each  factor 
also  has  a  direct  relationship  with  individual  learning,  the  strongest 
relationship  is  through  the  last  factor— worthwhile  investment  (B  = 
0.483). 

The  second  half  (right  side)  of  the  model  represents  how 
individual  learning  influences  job  performance  and  business  (orga¬ 
nizational)  outcomes.  By  examining  the  values,  we  see  from  the 
nondominant  (lowest  values)  paths  that  individual  learning  does 
not  have  a  strong  direct  effect  on  Job  Impact  and  Business  Results 
directly  (both  far  right)  60  days  after  training.  However,  by  follow¬ 
ing  the  dominant  paths,  we  see  that  Individual  Learning  leads  to 
Perceived  Future  Job  Impact,  actual  Job  Impact  in  60  Days  and 
Business  Results  in  60  Days.  This  path  indicates  that  knowledge 
for  the  sake  of  knowledge  (Newman,  1947)  is  not  sufficient.  Train¬ 
ing  must  be  perceived  as  relevant,  practical,  and  applicable  before 
learners  will  indicate  it  will  (and  does)  have  an  impact  on  perfor¬ 
mance  and  eventually  business  results.  The  entire  causal  pathway 
for  the  model  looks  like  a  giant  N,  starting  at  the  bottom  left,  rising 
to  the  top  left,  slanting  diagonally  to  the  bottom  right,  and  then  up 
to  the  top  right. 

Does  the  model  effectively  describe  the  relationship  among 
variables?  Yes,  the  model  has  a  relatively  high  explanatory  power 
for  predicting  Job  Impact  in  60  days  (R2  =  40.0%)  and  Business 
Results  in  60  days  (R2  =  59.2%).  A  model  that  predicts  the  structure 
and  relationships  perfectly  would  have  an  R2  value  of  100  percent, 
although  this  situation  is  virtually  impossible  to  achieve  in  social 
science  research.  To  put  this  measure  in  perspective,  consider  that 
it  is  an  algorithm  you  could  use  at  a  casino.  Given  the  value  of  the 
cards  in  your  hand  (e.g.,  Instructor  Effectiveness,  Courseware  Qual¬ 
ity,  etc.),  you  would  win  59.2  percent  of  the  time. 

This  predictive  learning  analytics  model  (KnowledgeAdvisors, 
Inc.)  is  relevant  to  DAU  because  it  serves  as  a  base  model  (derived 
from  a  data  set  of  over  1  million  surveys  industry-wide)  for  bench¬ 
marking  purposes.  As  a  result,  the  DAU  can  compare  its  model  and 
results  to  the  KnowledgeAdvisors’  predictive  learning  analytics 
model  to  determine  where  it  can  improve  and  where  it  is  outper¬ 
forming  other  organizations. 
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The  DAU  Model 

Each  year  the  DAU  collects  hundreds  of  thousands  of  evalu¬ 
ations  after  training  events  to  determine  whether  the  curriculum 
and  its  outcomes  were  effective.  Immediate  postcourse  evalua¬ 
tions  are  deployed  as  well  as  60-day  follow  up  evaluations.  In  this 
way,  learners  provide  feedback  about  the  quality  of  the  course  and 
predict  whether  they  will  apply  what  they  learned.  On  the  60-day 
follow  up  evaluation,  learners  indicate  whether  training  contributed 
to  improved  job  performance  and  business  results.  For  this  study 
more  than  326,000  evaluations  were  collected  during  19  months 
between  January  1,  2008,  and  July  30,  2009. 

Figure  2  shows  the  DAU  model  with  KnowledgeAdvisors  bench¬ 
mark  values  above  the  DAU  model  values.  The  analysis  reveals  many 
important  facts  about  the  DAU’s  curriculum. 

FIGURE  2.  PREDICTIVE  LEARNING  ANALYTICS  MODEL-DAU 

AND  SURVEY  INSTRUMENT  MODEL  BENCHMARK 


First,  the  causal  chain  depicted  in  the  model  explains  the  rela¬ 
tionships  among  the  data  well.  In  fact,  the  model  fits  the  DAU’s 
data  better  than  the  survey  instrument  benchmark  data.  (This  is  not 
unusual;  the  model  will  fit  some  data  sets  better  or  worse  than  the 
benchmark.)  The  model  predicts  52.7  percent  of  the  Job  Impact  in 
60  Days  and  67.9  percent  of  the  Business  Results  in  60  Days.  The 
values  for  the  KnowledgeAdvisors  Benchmark  data  are  lower  at 
40.0  percent  and  59.2  percent,  respectively.  These  results  indicate 
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that  the  key  aspects  of  training  that  drive  job  impact  and  business 
performance  hold  true  for  both  the  KnowledgeAdvisors  Benchmark 
data  and  the  DAU  data.  As  a  refresher,  Instructor  Effectiveness  links 
to  Courseware  Quality,  which  links  to  Worthwhile  Investment.  All 
three  in  that  order  optimize  Individual  Learning.  In  turn  Individual 
Learning  leads  to  Perceived  Future  Job  Impact,  Job  Impact  in  60 
Days,  and  then  Business  Results  in  60  Days  in  that  order.  To  improve 
the  effectiveness  of  courses— at  least  in  terms  of  increasing  job 
performance  and  business  impact— DAU  should  focus  on  Instructor 
Effectiveness,  Courseware  Quality,  and  ensure  that  learners  perceive 
that  training  is  a  Worthwhile  Investment.  Improvement  actions  will 
be  discussed  later  in  this  article. 

Second,  the  strength  of  the  causal  relationships  is  somewhat 
stronger  for  the  DAU  compared  to  the  KnowledgeAdvisors  Bench¬ 
mark  for  five  relationships  (arrows).  These  are  designated  with  a 
superscripted  a  after  the  value.  For  only  one  relationship,  the  link 
between  Courseware  Quality  and  Individual  Learning,  the  DAU  value 
is  lower  than  the  survey  instrument  benchmark  and  is  indicated  by  a 
superscripted  b.  When  this  relationship  was  examined  in  more  detail, 
it  was  discovered  that  Courseware  Quality  was  more  important  for 
younger  learners  and  less  so  for  older  learners.  Younger  learners 
preferred  e-learning,  whereas  older  learners  preferred  traditional 
classrooms  and  effective  instructors.  Interestingly,  younger  learn¬ 
ers  indicated  that  training  had  a  greater  impact  on  job  impact  and 
business  impact  60  days  after  training. 

Third,  DAU  instructors  have  a  strong  influence  on  Individual 
Learning  and  eventually  Job  Impact  and  Business  Results.  In  fact, 
when  compared  to  the  KnowledgeAdvisors  benchmark  (0.083), 
the  relationship  between  Instructor  Effectiveness  and  Individual 
Learning  (B  =  0.163)  is  almost  twice  as  large  for  DAU.  A  stronger 
relationship  between  Instructor  Effectiveness  and  Courseware  Qual¬ 
ity  still  exists,  but  by  comparing  the  magnitude  of  the  relationship 
between  the  DAU  and  the  benchmark,  clearly,  instructors  hold  more 
influence  within  DAU  than  at  other  organizations. 

Fourth,  guest  speakers  also  impact  learning.  When  guest  speak¬ 
ers  taught  courses,  higher  levels  of  Individual  Learning  occurred. 
When  guest  speakers  were  not  included,  Job  Impact  and  Business 
Results  were  generally  lower  than  the  survey  instrument  benchmark. 

Fifth,  application  is  a  critical  element  to  successful  courses.  H ig h 
job  application  scores  were  linked  to  high  learning  scores,  extremely 
high  Job  Impact  scores,  and  Business  Results  scores. 

Sixth,  application  is  also  strongly  linked  to  whether  learners  rec¬ 
ommend  courses  for  future  learners.  When  recommendation  scores 
were  low,  the  Business  Results  in  60  Days  were  also  lower.  This  is 
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a  strong  indicator  that  the  course  is  not  meeting  individual  needs 
and  organizational  needs  and  therefore  should  be  revised  or  retired. 

Seventh,  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  levels  were  investigated  to  test  for  their  influence  on 
training  outcomes.  DAWIA  specifies  three  skill  levels  (Basic,  Inter¬ 
mediate,  and  Advanced)  associated  with  13  career  fields  in  the 
acquisition  system.  No  consistent  pattern  of  influence  emerged 
across  the  model  for  the  three  DAWIA  levels. 

Eighth,  the  educational  level  of  learners  (e.g.,  high  school,  col¬ 
lege,  or  graduate  school)  does  influence  outcomes  of  the  model. 
Learners  with  some  graduate  education  are  more  critical  of  instruc¬ 
tors  and  appreciate  good  courseware.  Learners  with  a  high  school 
education  have  the  lowest  perception  that  training  is  a  Worthwhile 
Investment ,  but  yield  the  highest  response  to  Individual  Learning. 
Interestingly,  this  group  scored  much  lower  than  the  benchmark 
regarding  future  results. 

Lastly,  the  DAU’s  course  offerings  and  their  influence  on 
job  performance  and  outcomes  were  evaluated  for  longitudinal 
improvement.  Indeed,  it  was  confirmed  that  scores  improved  from 
2008  to  2009.  In  fact,  scores  for  every  category  improved  except 
for  Instructor  Effectiveness,  which  was  already  high. 


Recommended  Actions 

An  important  and  useful  finding  of  this  study  indicates  that 
the  key  aspects  of  training  drive  Job  Impact  and  Business  Results. 
This  in  itself  is  valuable,  but  such  value  quickly  fades  if  insights 
cannot  be  turned  into  action  to  improve  the  curriculum.  Table  2 
provides  a  summary  of  the  results  of  this  study  as  well  as  recom¬ 
mended  actions.  If  the  DAU  pursues  these  actions,  the  curriculum  is 
likely  to  improve  as  evidenced  by  improved  scores  on  the  training 
evaluations. 


Conclusions 

In  its  evolution,  DAU  has  broadly  embraced  adult  learning 
designs  in  its  formal  courses  and  accepted  the  fact  that  adults  learn 
best  “by  doing,”  whether  in  the  formal  learning  environment  or  on 
the  job.  With  formal  training,  DAU  attempts  to  “train  as  the  work¬ 
force  should  work,”  and  prepares  the  workforce  to  “work  as  they 
are  trained”  by  using  the  same  training  tools  and  learning  assets  at 
their  individual  places  of  work  that  they  formerly  used  in  the  class¬ 
room.  This  study  provides  strong  evidence  that  the  key  aspects  of 
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TABLE  2.  RECOMMENDED  ACTIONS 


Results 

Recommended  Action 

Application  is  a 

critical  element  of 

training 

As  appropriate,  DAU  can  improve  its 
impact  on  job  performance  and  business 
results  by  increasing  the  opportunities  to 
apply  what  is  learned  during  training. 

Courseware  Quality 
is  more  important  for 
younger  learners 

To  improve  learning  among  younger 
learners,  invest  in  self-study  modules  and 
quality  courseware. 

DAU  instructors  have 

a  strong  influence  on 

older  learners 

For  instructor-led  courses,  especially  with 
older  learners  as  the  target  audience, 
invest  time  and  effort  to  find  high-quality 
instructors  who  can  effectively  teach  the 
materials  regardless  of  the  quality  of  the 

courseware. 

Guest  speakers  also 
impact  learning 

When  appropriate,  use  guest  speakers  to 
augment  or  lead  instructor-led  courses 
for  older  learners.  Guest  speakers  tend 
to  have  more  impact  on  learning  than  the 

standard  cadre  of  instructors. 

Learners  recommend 

effective  training 

Use  the  question,  “1  would  recommend 
this  training  to  other  learners”  as  a  leading 
indicator  of  the  quality  of  training  and 
whether  it  will  lead  to  job  performance 
and  business  impact.  If  the  rating  for  this 
question  is  low  for  a  given  course,  it  should 
be  revised  with  a  focus  on  improving  the 
ability  to  apply  what  is  learned  during 
training. 

DAWIA  levels  do  not 

influence  training 
impacts 

When  building  DAU  courses,  it  is  not 
necessary  to  consider  the  DAWIA  level  of 
the  audience.  Other  factors  like  age  and 

education  are  more  influential  than  DAWIA 

levels. 

Education  impacts 
learning  and 
performance 

To  ensure  that  training  leads  to 
performance  and  future  results,  courses 

should  be  tailored  to  the  educational  level 

of  the  audience. 
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DAU’s  approach  to  training  drive  Job  Impact  and  Business  Results. 
Having  empirical  evidence  derived  from  relatively  large  data  sets  is 
very  useful  in  rationalizing  the  cost  of  training  regarding  improving 
performance  on  the  job.  Additionally,  confirming  the  reliability  and 
validity  of  the  survey  instrument  is  important  to  any  DAU  curricula 
and  recourse  decisions  based  on  survey  instrument  scores  as  well 
as  other  considerations. 

Given  that  Job  Impact  results  were  based  on  self-reporting 
perceptions  and  not  an  independent  external  measure  (not  within 
the  instrument),  the  outcome  is  still  very  strong  in  its  implications, 
largely  due  to  the  size  of  the  sample  as  well  as  previous  relation¬ 
ship  studies  concerning  the  close  relationships  between  measured 
perception  and  actual  reality.  Dess  and  Robinson  (1984)  indicate 
that  such  perceived  measures  of  business  results  are  reasonable 
surrogates  for  more  tangible  and  objective  measures  of  business 
outcomes  (e.g.,  revenue  growth,  profits).  Others  (Geringer  &  Hebert, 
1989;  Hansen  &  Wernerfelt,  1989;  Lyles  &  Salk,  1997;  Venkatra- 
man  &  Ramanujam,  1987)  have  demonstrated  that  such  perceived 
measures  also  are  positively  correlated  with  objective  financial 
performance  metrics. 

Finally,  the  recommendations  discussed  previously  are  now 
being  incorporated  within  DAU  course  development  and  course 
update  design  strategies.  Therein  lies  the  power  of:  (a)  bench¬ 
marking  learning  data  across  a  very  large  set  of  comparative  peer 
organizations;  and  (b)  using  structural  equation  modeling  to  ascer¬ 
tain  specific  points  of  intervention  for  evaluating  and  improving  the 
learning  enterprise,  thereby  assuring  a  healthy  return  on  investment 
for  training  dollars. 
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ENDNOTES 

1.  LISREL,  an  acronym  for  linear  structural  relations,  is  a  statistical  software  package  used 
in  structural  equation  modeling.  LISREL  was  developed  in  the  1970s  by  Karl  Joreskog, 
then  a  scientist  at  Educational  Testing  Service  in  Princeton,  NJ,  and  Dag  Sorbom,  later 
both  professors  of  Uppsala  University,  Sweden. 

2.  AMOS,  an  acronym  for  analysis  of  moment  structures,  is  designed  primarily  for  structural 
equation  modeling,  path  analysis,  and  covariance  structure  modeling,  though  it  may  be 
used  to  perform  linear  regression  analysis.  It  features  an  intuitive  graphical  interface  that 
allows  the  analyst  to  specify  models  by  drawing  them.  It  also  has  a  built-in  bootstrapping 
routine  and  superior  handling  of  missing  data.  It  reads  data  from  a  number  of  sources, 
including  MS  Excel®  spreadsheets. 
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MOTIVATING  THE 
KNOWLEDGE  WORKER 

i  David  E.  Frick 

Commonly  accepted  economic  theory  suggests  that  workers 
are  rational  actors  and  make  decisions  that  will  maximize 
expected  outcomes.  As  such,  managers  should  be  able  to 
influence  behaviors  to  meet  business  goals  by  manipulating 
the  expectations  of  outcomes.  Conversely,  social  science 
practitioners  suggest  that  workers  often  make  decisions  that 
are  irrational.  Knowledge  workers  are  a  growing  sector  of  the 
workforce  and  are  the  backbone  for  entire  federal  agencies. 

The  acquisition  community  falls  within  this  category.  Identifying 
factors  that  influence  the  performance  of  knowledge  workers 
may  be  critical  to  maintaining  high  levels  of  organizational 
performance.  This  research  focused  on  identifying  the  factors 
that  encourage  knowledge  workers  to  maintain  high  levels  of 
performance. 


Keywords:  Motivation,  Knowledge  Worker, 
Performance,  Human  Capital,  Merit  Systems 
Protection  Board 


image  designed  by  Tia  Gray  » 


Positive 


Neg 

MOTIVATING 

FACTORS 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


0<XXX><><X>0<Xx>00<><X><><><><X><>500<>0<><>0<><><>0<>0<>0000<^ 

"There  is  one  right  way  to  manage  people— or  at 

LEAST  THERE  SHOULD  BE." 

-Peter  F.  Drucker 

oo<><>d<><><>o<>o<><x><><><>o<x><x><><><>^ 

Conventional  wisdom  and  commonly  accepted  economic  theory 
suggest  that  workers  are  rational  actors  and  make  decisions  that 
will  maximize  expected  outcomes— maximize  expected  benefits 
or  minimize  expected  harm.  As  such,  managers  should  be  able  to 
influence  behaviors  to  meet  business  goals  by  manipulating  the 
expectations  of  outcomes.  Etzioni  (1971)  argues  that  workers  find 
this  manipulation  of  behavior  via  incentives  alienating  and  dehu¬ 
manizing.  Conversely,  social  science  practitioners  suggest  that 
workers  often  make  decisions  that  are  irrational  (from  an  economic 
perspective)  and  are  based  on  cognitive  biases  (Santaniello,  2008). 
These  beliefs  have  been  formed  over  the  last  100  years  in  an  envi¬ 
ronment  that  has  been  dominated  by  agricultural,  manufacturing, 
and  industrial  workers. 

Knowledge  workers  are  a  growing  sector  of  the  workforce 
(Haag,  Cummings,  &  Phillips,  2008).  They  are  individuals  valued  for 
their  ability  to  gather,  analyze,  interpret,  and  synthesize  information 
within  specific  subject  areas  to  advance  the  overall  understanding 
of  those  areas  and  allow  organizations  to  make  better  decisions. 
The  knowledge  worker  is  the  backbone  of  many  professions.  Within 
the  federal  government,  entire  agencies  are  comprised  mainly  of 
knowledge  workers.  The  members  of  the  acquisition  community 
principally  fall  within  this  definition. 

Creating  environments  to  encourage  high  performance  among 
knowledge  workers  is  an  area  long  neglected  by  researchers.  To 
date,  no  published  research  exists  on  knowledge  workers  in  the 
federal  government.  Even  the  term  knowledge  worker  was  not 
defined  until  1999  (Drucker,  1999).  As  a  consequence  of  this  lack  of 
evidence,  the  executive  branch  has  been  forced  into  making  stra¬ 
tegic  human  capital  decisions  based  upon  theory  and  experiences 
that  may  not  apply  to  the  knowledge  worker. 

Collins  (2001)  looked  at  high-performing  companies  to  see  if  he 
could  find  patterns  within  the  cultures  of  the  respective  workforces. 
His  methodology  was  questionable  and  his  conclusions  were  not 
particularly  useful,  but  he  did  make  two  statements  that  are  quite 
provocative:  “...expending  energy  trying  to  motivate  people  is  largely 
a  waste  of  time”  (p.  74)  and  “You  cannot  manufacture  passion  or 
‘motivate’  people  to  feel  passionate.  You  can  only  discover  what 
ignites  your  passion  and  the  passion  of  those  around  you”  (p.  109). 

Public  Service  Motivation  Theory  (Crewson,  1997;  Houston, 
2000;  Perry,  1996;  Porter  &  Perry,  1982;  Perry  &  Wise,  1990)  sug¬ 
gests  that  individuals  who  self-select  into  government  service  are 
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motivated  by  a  set  of  factors  (self-sacrifice,  desire  to  serve  the  pub¬ 
lic,  desire  to  serve  a  higher  power)  that  is  more  intrinsically  centered 
than  the  set  of  factors  that  motivates  private  sector  workers.  Small 
contingency-based  rewards,  such  as  the  insubstantial  pay  increases 
common  in  government  pay-for-performance  systems,  tend  to 
crowd  out  these  intrinsic  factors. 

Public  administration  literature  also  makes  a  distinction  between 
employee  motives  and  work  motivation.  Motives  are  the  rewards 
that  workers  would  like  to  receive  for  their  jobs,  while  work  motiva¬ 
tion  is  defined  as  the  drive  workers  have  to  perform  their  jobs  well 
within  the  rewards  offered  by  the  government  and  private  sectors. 
Workers  self-select  into  either  the  public  or  private  sector  based 
on  whether  the  incentive  structure  is  aligned  with  their  individual 
values  and  motives  (Rainey,  1982). 

A  significant  weakness  in  the  civil  service  is  the  inability,  in 
practice,  of  managers  to  weed  out  inferior  performers.  In  2008,  the 
federal  government  only  fired  11,165  employees  (0.57  percent  of 
the  workforce)  (Losey,  2009).  Compare  this  with  the  Merit  Systems 
Protection  Board  (MSPB)  estimate  that  3.7  percent  of  the  federal 
workforce  are  poor  performers  (Office  of  Personnel  Management 
[OPM],  1999,  p.  1).  For  business,  especially  those  in  “employment 
at  will”  states,  the  process  of  eliminating  substandard  performers 
is  significantly  less  arduous  than  in  the  civil  service.  The  danger  of 
frivolous  claims  of  discrimination  always  remains,  but  on  balance, 
business  has  a  flexibility  that  the  government  does  not  have... in 
practice.  Yes,  the  civil  service  rules  do  allow  for  removing  nonper¬ 
formers,  but  the  process  is  labor-intensive  for  supervisors,  extremely 
drawn  out,  and  subject  to  a  number  of  administrative  reviews  that 
tend  to  encourage  supervisors  to  use  an  alternate  method  for 
eliminating  inferior  performers— in  other  words,  “passing  the  trash” 
(Shuger,  1999). 

Workforce  mobility  in  the  civil  service  is  rooted  in  the  “Peter 
Principle”— primarily  centered  on  upward  mobility.  The  way  to  get 
promoted  is  to  find  a  job  vacancy  at  a  higher  grade  and  compete 
against  other  applicants.  In  most  hiring  processes,  performance 
evaluations  are  a  consideration  in  the  hiring  decision.  Some  low- 
performing  supervisors  have  been  known  to  artificially  inflate  the 
performance  evaluations  of  inferior  supporters  with  the  goal  of 
passing  the  trash  to  someone  else  (Shuger,  1999;  Peter  &  Hull,  1969). 
Yes,  this  is  unfortunate  and  not  in  the  best  interest  of  the  public 
good,  but  it  happens  at  all  levels. 

Additionally,  the  civil  service  exhibits  a  characteristic  that  many 
workers  find  invaluable,  especially  Baby  Boomers  and,  to  a  lesser 
extent,  “Gen  X’ers”— job  security  (Alsop,  2008).  Job  security  has  a 
value  (Crewson,  1997;  Houston,  2000).  It  is  reasonable  to  concede 
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to  the  employer,  the  taxpayer  in  the  case  of  civil  service,  something 
of  equal  benefit  for  the  benefit  of  job  security.  Historically,  that  has 
been  a  pay  structure  that  has,  arguably,  lagged  behind  the  free 
market.  Job  security  is  the  tradeoff  for  a  lagging  pay  policy.  Pay  for 
performance  appears  to  be  a  technique  to  solve  the  “lagging  pay” 
issue  that  does  not  have  an  equitable  tradeoff  for  the  employer  (the 
people  of  the  United  States). 


Purpose  of  the  Study 

The  author  designed  a  study  to  gather  the  opinions  of  a  group  of 
independently  identified,  high-performing  federal  civilian  employ¬ 
ees  from  multiple  agencies  to  develop  a  rank-ordered  list  of  factors 
that  may  be  most  effective  in  establishing  an  environment  that  moti¬ 
vates  high-performing  knowledge  workers  to  maintain  high  levels 
of  performance.  All  participants  in  this  survey  appear  to  meet  the 
definition  of  knowledge  worker. 


Theoretical  Framework 

No  universally  accepted  model  of  motivation  is  inherent  to  a 
business  environment.  To  facilitate  a  structured  approach  to  the 
analysis  of  the  various  theories  of  motivation  and  the  data  collected 
from  this  effort,  the  author  developed  a  two-dimensional  model  of 
the  factors  that  motivate  workers  (Table  1). 


TABLE  1.  FRAMEWORK  AXES 

Factor 

Description 

Logical 

Associated  with  a  process  either  inductive 

or  deductive.  Elements  tend  to  be  more 

tangible  than  intangible.  Cognitive. 

Emotional 

Associated  with  responses  that  are  based 
upon  intuition,  prior  learning,  perceptions, 

and  desires. 

Controlled 

Elements,  decisions,  or  expectations  can  be 
formed  by  the  individual.  The  world  tends 
to  be  defined  by  internal  filters. 

Uncontrolled 

Elements,  influences,  conditions,  and 
constraints  are  established  by  either  the 

environment  or  an  outside  actor.  The  world 

tends  to  be  defined  by  external  filters. 
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The  horizontal  axis  represents  the  universe  from  a  contextual 
viewpoint.  The  vertical  axis  takes  the  content  viewpoint.  Context, 
from  the  viewpoint  of  the  worker,  can  be  either  controllable  or 
uncontrollable.  Content  is  either  logical  (tangible-cognitive)  or 
emotional  (intangible-instinctual).  The  four  quadrants  represent 
environmental  and  hygiene  factors,  contingent  rewards,  and  rela¬ 
tionships.  The  central  area  is  reserved  for  those  theories  or  factors 
that  have  mixed  characteristics  or  do  not  clearly  fit  into  a  quadrant 
(Figure). 

FIGURE.  THEORETICAL  FRAMEWORK 
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Significance  of  the  Study 

As  the  U.S.  economy  and  those  government  agencies  that 
support  the  nation’s  institutions  become  more  dependent  upon 
knowledge  workers,  the  need  to  fully  understand  those  conditions 
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that  encourage  continued  high  performance  among  knowledge 
workers  becomes  more  important. 

This  research  identified  a  set  of  conditions  that  are  effective  in 
cultivating  a  state  of  positive  motivation  among  knowledge  work¬ 
ers  of  the  federal  workforce.  Identifying  those  factors  that  this  one 
specific  subset  of  the  workforce  believes  are  most  motivating  may 
provide  strategic  leaders  with  the  empirical  information  needed  to 
make  more  effective  decisions.  Likewise,  allowing  managers  to  more 
effectively  commit  organizational  resources  will  further  the  goal  of 
improving  overall  performance  of  the  entire  workforce. 


Method— Highlights 

The  target  population  was  the  2009  Fellowship  of  the  Council 
for  Excellence  in  Government  (CEG)  of  the  Partnership  for  Public 
Service.  This  population,  which  represented  a  high-performing 
subset  of  the  federal  workforce,  was  selected  for  convenience.  A 
precise  definition  of,  and  contact  information  for,  the  entire  popula¬ 
tion  was  available  to  the  author. 

There  were  132  federal  workers  in  the  population.  The  sample 
was  self-selected.  All  subjects  were  volunteers.  Sixty-four  members 
of  the  cohort  agreed  to  participate  in  the  study.  The  sample  suffered 
from  self-selection  bias. 

A  survey  instrument  was  created  for  this  study.  A  wide  range  of 
structured,  demographic  information  was  collected.  Opinion  ques¬ 
tions  were  open-ended,  but  included  a  forced  distribution  system. 
A  number  of  uncommon  demographic  categories  were  included 
in  the  hope  of  serendipitous  findings  and  to  ascertain  whether  the 
sample  was  similar  to  the  entire  federal  workforce  or  the  general 
population. 

The  author  was  able  to  infer  that  the  subjects  were  high- 
performing  by  their  participation  in  the  highly  competitive  CEG 
program.  The  cost  to  the  agencies  to  participate  was  $10,000  per 
participant.  Replicating  this  study  by  assembling  a  similar  group 
comprised  only  of  high-performing  workers  in  any  other  environ¬ 
ment  would  prove  difficult. 

While  this  study  may  show  clear  preferences  of  the  workforce, 
it  was  not  able  to  show  a  causal  relationship  between  identified 
factors  and  workforce  performance. 

Questions  self-report  preferences;  therefore,  problems  of  self- 
report  bias  need  to  be  taken  into  account,  as  responses  may  not 
be  completely  accurate  (Rosenthal  &  Rosnow,  1991).  Additionally, 
appropriate  group  norms  in  this  research  area  to  interpret  measures 
were  not  available. 
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In-depth  validity  and  reliability  studies  of  the  survey  instruments 
were  not  conducted. 

The  sample  used  in  this  study  was  relatively  small  and  admit¬ 
tedly  atypical  of  the  entire  federal  workforce.  Generalization  to  all 
federal  knowledge  workers  or  the  acquisition  workforce,  in  particu¬ 
lar,  was  not  possible. 


Knowledge  Worker 

Knowledge  workers  are  generally  professionals  such  as  teach¬ 
ers,  lawyers,  architects,  physicians,  nurses,  engineers,  and  scientists. 
As  businesses  increase  their  dependence  on  information  technol¬ 
ogy,  the  number  of  fields  in  which  knowledge  workers  must  operate 
has  expanded  dramatically. 

The  term  was  first  coined  by  Peter  Drucker  in  1959,  and  later 
refined  in  1999,  as  one  who  works  primarily  with  information  or  one 
who  develops  and  uses  knowledge  in  the  workplace  (Drucker,  1973, 
1999).  Some  tasks  that  are  performed  by  the  acquisition  community 
do  not  fall  within  the  definition  of  knowledge  work;  however,  those 
aspects  that  involve  making  judgments  and  trade-off  decisions 
clearly  do. 

Drucker  (2001)  added  to  the  definition  of  knowledge  workers 
by  describing  their  fundamental  tasks. 

To  be  sure,  the  fundamental  task  of  management  remains  the 
same:  to  make  people  capable  of  joint  performance  through 
common  goals,  common  values,  the  right  structure,  and  the 
training  and  development  they  need  to  perform  and  to  respond 
to  change.  But  the  very  meaning  of  this  task  has  changed,  if 
only  because  the  performance  of  management  has  converted 
the  workforce  from  one  composed  largely  of  unskilled  laborers 
to  one  of  highly  educated  knowledge  workers,  (p.  4) 

Even  if  employed  full-time  by  the  organization,  fewer  and  fewer 
people  are  ‘subordinates’— even  in  fairly  low-level  jobs.  Increas¬ 
ingly  they  are  ‘knowledge  workers.’  And  knowledge  workers  are 
not  subordinates;  they  are  ‘associates.’  For,  once  beyond  the 
apprentice  stage,  knowledge  workers  must  know  more  about 
their  job  than  their  boss  does— or  else  they  are  no  good  at  all. 
In  fact,  that  they  know  more  about  their  job  than  anybody  else 
in  the  organization  is  part  of  the  definition  of  knowledge  work¬ 
ers.  (p.  78) 
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Literature 

How  do  the  best  managers  in  the  world  build  the  foundation  for 
a  strong,  high-performing  workplace?  No  clear  answer  can  be  found. 
In  1975,  200  books  were  published  on  the  topic  of  management  and 
leadership.  By  1997,  that  number  had  tripled  (Buckingham  &  Coff¬ 
man,  p.  53).  A  quick  search  today  of  the  website  Amazon.com  with 
the  keywords  “management  and  leadership”  yields  over  350,000 
results. 

With  respect  to  people,  Reiss  (2000)  suggests  that  there  are  16 
distinct  basic  desires  that  “make  our  lives  meaningful.”  He  claims 
that  everyone  displays  each  of  these  desires  either  strongly,  mod¬ 
erately,  or  weakly.  If  his  hypothesis  is  valid,  then  there  are  a  possible 
43,046,721  distinct  possible  personality  types,  while  Myers  and 
Briggs  claim  16  distinct  personality  types  (Myers  &  Myers,  1995). 

A  little  closer  to  home,  my  own  mother  unhesitatingly  classified 
workers  into  two  distinct  personality  preferences.  "There  are  two 
kinds  of  people— those  who  do  the  work  and  those  who  take  the 
credit,  ”  she  would  often  say.  She  did  not  realize  she  was  quoting 
Indira  Gandhi,  who  went  on  to  add,  "Try  to  be  in  the  first  group; 
there  is  less  competition  there.” 

But  whether  there  are  2, 16,  or  43  million  different  types  of  peo¬ 
ple,  finding  a  single  model  to  portray  how  all  people  are  motivated 
has  proven  to  be  extremely  difficult. 


Theories  of  Motivation 

Business  is  constantly  looking  for  the  best  practice  and  often 
engages  an  expert  to  demonstrate  the  one  best  way.  Drucker  (2001) 
often  spoke  of  the  futility  of  management  practitioners  in  finding 
the  one  right  theory  or  the  one  right  way  to  manage,  and  the  belief 
that  one  exists. 

Basic  assumptions  about  reality  are  the  paradigms  of  a  social  sci¬ 
ence  such  as  management.  They  are  usually  held  subconsciously 
by  the  scholars,  the  writers,  the  teachers,  the  practitioners  in 
the  field,  and  are  incorporated  into  the  discipline  by  their  vari¬ 
ous  formulations.  Thus,  those  assumptions  by  this  select  group 
of  people  largely  determine  what  the  discipline  assumes  to  be 
reality,  (p.  69) 

These  assumptions  underlie  practically  every  book  or  paper  on 
the  management  of  people. 
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Regrettably,  in  motivating  workers,  there  is  no  best  practice ; 
there  is  no  Grand  Theory  on  Motivation ;  there  is  no  commonly 
accepted  model  of  motivation. 

At  least  34  principal  theories  of  motivation  have  some  appli¬ 
cation  to  business.  Most  are  contradictory.  The  reason  is  simple... 
people  are  complex  animals.  Business  practitioners  have  long 
searched  for  the  one  right  business  model,  the  one  right  organiza¬ 
tional  structure,  the  one  right  management  style,  and  the  one  right 
way  to  treat  employees  (Drucker,  2001).  So  far  that  search  has  been 
in  vain. 


One  way  to  look  at  these  theories  is  to  classify  them  by  moti¬ 
vating  factors  from  the  perspective  of  the  worker.  These  factors 
represent  the  worker’s  preference,  e.g.,  for  a  given  worker,  contin¬ 
gent  rewards  might  be  more  effective  in  achieving  desired  behaviors 
than  personal  relationships.  The  figure  represents  the  author’s  inter¬ 
pretation  of  each  of  these  theories  with  respect  to  the  framework. 

An  examination  of  all  of  these  theories  is  well  beyond  the  scope 
of  this  article.  Nonetheless,  the  following  discussion  focuses  on 
motivational  theorists  of  some  note. 

Theorists  in  the  Business  Environment 

The  theories  of  Maslow,  McGregor,  and  to  a  lesser  extent,  Her- 
zberg,  are  the  most  commonly  accepted  in  business  literature.  At 
lower  levels  of  Maslow’s  hierarchy  of  needs,  such  as  physiological 
needs,  money  is  a  motivator;  however,  it  tends  to  have  a  motivating 
effect  on  employees  that  lasts  only  for  a  short  period.  At  higher  lev- 
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els  of  the  hierarchy,  praise,  respect,  recognition,  empowerment,  and 
a  sense  of  belonging  are  far  more  powerful  motivators  than  money. 

McGregor  (1960)  asserted  that  management  must  choose 
between  two  and  only  two  different  ways  of  managing  people— 
’’Theory  X”  and  “Theory  Y”— and  then  asserted  that  Theory  Y  is 
the  only  sound  one.  Drucker  (2001,  p.  77)  points  out  a  few  years 
later  that  Maslow  suggested  in  his  Eupsychian  Management  (1965), 
republished  as  Maslow  on  Management  (1998),  that  McGregor  was 
wrong.  He  showed  conclusively  that  “different  people  have  to  be 
managed  differently.” 

Herzberg  differentiated  hygiene  factors  from  motivators  in  the 
length  of  time  the  particular  factor  continues  to  drive  behaviors. 
Salary  (base  pay)  has  a  short  motivational  time  span.  “An  employee 
might  receive  a  pay  raise  today,  and  30  days  later  begin  to  question 
when  the  next  raise  will  be  forthcoming.  Meanwhile,  the  current 
salary  has  little  influence  on. ..willingness  to  improve  performance” 
(Henderson,  2002,  p.  391). 

Maslow  has  money  at  the  lowest  level  of  the  hierarchy  and  shows 
other  needs  are  better  motivators  to  employees.  McGregor  places 
money  in  his  Theory  X  category  and  considers  it  as  a  poor  motiva¬ 
tor.  Praise  and  recognition  are  placed  in  the  Theory  Y  category  and 
are  considered  stronger  motivators  than  money.  Likewise,  McClel¬ 
land  (1987)  noted  that  workers  could  not  be  motivated  by  the  mere 
need  for  money— in  fact,  extrinsic  motivation  (e.g.,  money)  could 
extinguish  intrinsic  motivation  such  as  achievement  motivation, 
though  money  could  be  used  as  an  indicator  of  success  for  various 
motives,  e.g.,  keeping  score. 

In  The  Peter  Principle  (Peter  &  Hull,  1969),  we  were  warned 
that  if  managers  follow  the  path  of  conventional  wisdom  without 
question,  they  tend  to  promote  each  person  to  his  or  her  level  of 
incompetence.  It  was  true  then,  and  it  is  true  now. 

Buckingham  and  Coffman  (1999)  note  little  or  no  change  to  this 
conventional  wisdom. 

Unfortunately,  in  the  intervening  years,  we  haven’t  succeeded  in 
changing  very  much.  We  still  think  that  the  most  creative  way  to 
reward  excellence  in  a  role  is  to  promote  the  person  out  of  it.  We 
still  tie  pay,  perks,  and  titles  to  a  rung  on  the  ladder:  the  higher 
the  rung,  the  greater  the  pay;  the  better  the  perks,  the  grander 
the  title.  Every  signal  we  send  tells  the  employee  to  look  inward 
and  upward.  ‘Don’t  stay  in  your  current  role  for  too  long,’  we 
advise.  ‘It  looks  bad  on  the  resume.  Keep  pressing,  pushing,  and 
stretching  to  take  that  next  step.  It’s  the  only  way  to  get  ahead. 
It’s  the  only  way  to  get  respect.’  (p.  178) 
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Public  Service  Motivation 

“The  theory  of  public  service  motivation  (PSM)  suggests  public 
employees  are  more  likely  than  private  sector  employees  to  hold 
pro-social  values  and  seek  opportunities  to  help  others  benefit 
society”  (Wright,  2007,  p.  5).  The  term  seems  to  have  been  coined 
by  Perry  and  Wise  (1990,  p.  368). 

“The  public  administration  literature  argues  that  individuals 
employed  by  the  government  have  a  unique  sense  of  public  service 
that  leads  them  to  value  intrinsic  rewards  more  keenly  than  extrinsic 
rewards,  although  few  studies  have  investigated  the  concept  empiri¬ 
cally”  (Santaniello,  2008,  p.  1). 

According  to  Perry  (1996,  p.  6),  PSM  is  defined  as  “an  indi¬ 
vidual’s  predisposition  to  respond  to  motives  grounded  primarily 
or  uniquely  in  public  institutions  and  organizations.”  Attraction  to 
policy  making,  commitment  to  the  public  interest,  compassion, 
and  self-sacrifice  are  all  identified  as  key  components  of  PSM  as 
opposed  to  extrinsic  motivations  (specifically  pay)  that  are  central 
to  many  rational  choice  models  of  motivation  and  provide  the  basis 
for  pay-for-performance  structures.  This  theory  contradicts  the 
conventional  wisdom  of  economic  theory  (Santaniello,  2008,  p.  3). 

Public  administration  literature  makes  a  distinction  between 
employee  motives  and  work  motivation.  Motives  are  the  rewards 
that  individual  employees  would  like  to  receive  for  their  jobs,  while 
work  motivation  is  the  drive  employees  have  to  perform  their  jobs 
well  within  the  context  of  their  organizations  (Wright,  2007). 


Poor  Performers  in  the  Federal  Government 

The  best  estimate  of  the  proportion  of  poor  performers  in  the 
federal  workforce  is  3.7  percent.  While  no  good  benchmarks  exist  in 
the  private  or  public  sectors,  such  comparison  is  undoubtedly  lower 
than  conventional  wisdom.  Supposedly,  the  federal  government  has 
no  serious  performance  problems  (OPM,  1999,  p.  1). 

Nonetheless,  the  prevailing  perception  about  public  service 
employment  is  that  poor  performance  is  a  big  problem.  Moreover, 
civil  service  employees  are  among  the  first  to  speak  up  about  the 
situation.  In  questionnaire  after  questionnaire,  civil  service  employ¬ 
ees  express  disdain  for  a  management  team  that  they  say  cannot 
or  will  not  remove  from  their  midst  coworkers  who  are  not  carrying 
their  share  of  the  load.  In  a  1997  report  entitled  Adherence  to  the 
Merit  Principles  in  the  Workplace,  the  MSPB  reported  that,  among 
the  9,700  federal  employees  it  surveyed,  the  issue  of  handling  poor 
performance  was  the  deepest  area  of  concern.  Nearly  half  of  the 
respondents  said  that  agencies  had  a  major  problem  correcting 
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poor  performance,  and  even  more  said  the  same  thing  about  the 
firing  of  poor  performers  (OPM,  1999,  p.  3). 

As  part  of  its  Fiscal  Year  1998  oversight  program,  the  OPM 
examined  the  foundation  for  the  suspicion  that  the  government 
has  too  many  poor  performers.  To  do  this,  they  identified  a  random 
sample  of  employees  and  interviewed  their  supervisors.  For  those 
employees  identified  as  “poor  performers,”  the  supervisors  were 
asked  what  caused  the  employee’s  poor  performance,  what  was 
done  to  address  the  problem,  and  what  had  been  achieved  as  a 
result.  In  a  separate  interview  sample,  OPM  contacted  supervisors 
who  had  successfully  taken  a  formal  action  to  deal  with  a  poor  per¬ 
former,  to  obtain  descriptions  of  their  experiences,  and  to  record 
the  lessons  they  learned.  Finally,  OPM  looked  at  the  private  sector 
and  public  sector  for  points  of  comparison.  The  implication  is  that 
these  problems  are  less  prevalent  outside  the  federal  government 
(OPM,  1999,  p.  4). 

To  exacerbate  this  prevailing  perception,  surveys  find  that  most 
federal  workers  do  not  believe  that  the  best  qualified  people  are 
the  ones  receiving  promotions  (MSPB,  2001,  p.  7).  Sometimes,  the 
motivation  to  retain  poor  performers  is  rooted  in  the  federal  hiring 
process.  The  White  House  has  noted  that  it  can  often  take  18  months 
or  longer  to  fire  employees,  thus  requiring  a  major  commitment  of 
time  and  effort  from  managers  (Edwards  &  DeHaven,  2002,  p.  2). 

Most  managers  try  to  work  around  bad  employees  or  try  to 
reassign  them  to  other  groups.  OPM  surveys  consistently  find  that 
managers  think  that  “procedures  dealing  with  poor  performance 
are  too  complicated,  time  consuming,  or  onerous;  they  do  not  get 
higher  management  support;  and  they  perceive  their  decisions  will 
be  reversed  or  that  they  will  be  falsely  accused  of  discrimination  in 
their  actions”  (OPM,  1999,  p.  1).  Those  fears  are  justified  given  that 
federal  workers  lodge  discrimination  complaints  at  10  times  the  rate 
of  nonfederal  workers  (OPM,  1999,  pp.  3, 11). 

Another  problem  is  that  poor  performers  often  receive  good 
performance  reviews  from  negligent  managers  who  do  not  want  to 
rock  the  boat.  There  is  an  ingrained  federal  culture  to  score  virtu¬ 
ally  all  workers  highly— the  MSPB  has  found  that  just  1  percent  of 
federal  workers  are  rated  below  “fully  successful”  in  annual  reviews 
(MSPB,  1999,  p.  12). 

The  various  theories  of  motivation  are  too  contradictory  to  sug¬ 
gest  the  “one  right  answer.”  The  search  for  the  one  or  two  best  ways 
to  manage  people  has  been  in  vain.  When  you  add  human  nature 
with  its  myriad  cognitive  biases  to  the  mix,  management  action 
often  leads  to  outcomes  that  have  a  net  negative  effect.  The  lack  of 
empirical  evidence  to  paint  a  clear  picture  of  the  emerging  knowl¬ 
edge  worker  makes  the  challenges  of  picking  the  right  solutions 
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even  harder.  Understandably,  business  and  government  leaders  of 
today  may  make  strategic  decisions  that,  only  in  hindsight,  prove 
to  be  ineffective. 


Results  of  the  Study 


The  following  tables  TABLE  2.  POSITIVE  FACTORS 

reflect  the  positive  (Table  2) 
and  negative  factors 
(Table  3)  as  reported  by 
the  sample.  Common  defi¬ 
nitions  are  assumed  for  all 
terms,  although  respon¬ 
dents  perhaps  had  differing 
understandings  of  these 
common  terms,  e.g.,  insuf¬ 
ficient  resources  could  mean 
budget  shortfalls,  person¬ 
nel  shortages,  insufficient 
physical  facilities,  or  a  com¬ 
bination  of  all. 

These  results  are  con¬ 
sistent  with  similar  studies 
that  looked  specifically  at 
work  compensation.  Total 
work  compensation  has  an 
influence  on  worker  motiva¬ 
tion,  but  it  is  not  a  significant 
factor  affecting  the  behav¬ 
iors  that  lead  to  measures  of 
performance  for  knowledge 

workers  in  the  federal  workforce.  The  factors  that  are  most  influen¬ 
tial  are  intangible,  emotion-based,  and  intrinsic.  The  top  5  positive 
factors— meaningful  work,  belief  in  mission,  sense  of  public  service, 
opportunity  to  advance,  and  relationship  with  coworkers— are  all 
highly  personal  and  defined  by  the  individual  (Table  2).  Conversely, 
the  five  most  influential  negative  factors— insufficient  resources,  the 
“bad  manager,”  a  perception  of  a  lack  of  support  from  managers, 
an  unwillingness  to  deal  with  substandard  performers,  and  the  dif¬ 
ficulty  of  the  daily  commute  (Table  3)— are  principally  influenced, 
if  not  defined,  by  external  actors. 


Rank 

Top  Positive  Factors 

l 

Meaningful  work 

2 

Belief  in  mission 

3 

Public  service 

4 

Opportunity  to  advance 

5 

Relationship  with  coworkers 

6 

Relationship  with  supervisor 

7 

Personal  work  ethic 

8 

Education  benefits 

9 

Great  people 

10 

Flexible  workplace  policy 

11 

Empowerment 

12 

Organizational  values 

13 

Teamwork 

14 

Supportive  management 

14 

Recognition  by  others 

16 

Total  compensation 

17 

Equitable  awards  policy 

18 

Job  security 
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TABLE  3.  NEGATIVE  FACTORS 


Rank 

Top  Negative  Factors 

i 

Insufficient  resources 

2 

Bad  managers 

3 

Lack  of  management  support 

4 

Unwillingness  to  deal  with  substandard  performers 

5 

Difficult  commute 

6 

Ineffective  technology 

7 

Lack  of  planning 

8 

Lazy  coworkers 

9 

Abusive  supervisors 

10 

Lack  of  teamwork 

11 

Lack  of  promotion  opportunities 

12 

Corruption  in  the  workplace 

13 

Management  resistance  to  change 

14 

Negative  organizational  culture 

Conclusions 

This  group  of  federal  employees  expressed  a  preference  for 
intrinsic  (internal)  factors.  This  is  consistent  with  PSM  Theory.  Con¬ 
ditions  that  have  the  greatest  negative  effect  appear  to  be  those  in 
which  the  workers  have  no  direct  control,  e.g.,  how  managers  deal 
with  substandard  performers,  the  quality  of  supervisors,  and  the 
sufficiency  of  resources. 

Implications 

Regrettably,  this  study  does  not  help  Drucker’s  search  for  “one 
right  way  to  manage  people.”  As  previously  discussed,  emotional- 
uncontrolled  (contingent  rewards)  and  mixed  theories  appear  to  be 
the  most  relevant.  This  was  most  surprising  as  these  two  sections 
encompass  relatively  few  of  the  major  theories.  Even  more  interest¬ 
ing,  was  the  section  logical-controlled  (hygiene).  In  the  model,  this 
section  includes  the  greatest  number  of  discrete  theories,  which 
appear  to  have  the  least  relevance.  This  result  was  unexpected  by 
the  author,  who  expected  factors  consistent  with  Equity  Theory  to 
dominate.  However,  the  expected  key  words,  e.g.,  fairness,  equality, 
justice,  deserved,  were  scarce  in  the  responses.  This  suggests  that 
the  subjects  believe  that  either  the  current  environment  is  equitable 
and  does  not  influence  their  performance  or  that  equitability  is  not 
a  significant  factor. 
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The  results  of  this  study  also  support  the  assertion  that  Maslow’s 
Hierarchy  of  Needs,  Herzberg’s  Two-factor  Theory,  and  McGregor’s 
Theory  X/Theory  Y  are  highly  relevant  to  this  population  and  to 
business  in  general. 

Since  employment  satisfies  physiological  and  safety 
needs,  love/belonging  (relationships  with  coworkers,  relationship 
with  supervisors),  esteem  (opportunity  to  advance,  empowerment) 
and  self-actualization  (belief  in  mission,  meaningful  work)  become 
more  important. 

Herzberg,  The  Two-factor  Theory  asserts  that  motivators  and 
de-motivators  are  mutually  exclusive  sets  of  factors.  This  research 
supports  this  assertion.  A  weak  argument  can  be  made  that  some 
of  the  factors  are  not  true  opposites,  but  are  strongly  related, 
e.g.,  great  people— lazy  coworkers,  relationship  with  supervisors— 
bad  managers;  however,  further  investigation  would  be  needed  to 
support  this  argument. 

McGregor.  These  findings  suggest  that  public  sector  knowledge 
workers  are  self-motivated  and  will  perform  at  the  highest  possible 
level  when  barriers  to  performance  are  absent.  This  is  consistent 
with  the  Theory  Y  assertion  that  employees  will  seek  out  and  accept 
responsibility,  exercise  self-control  and  self-direction,  and  will  work 
well  given  the  right  conditions. 

Implications  for  Practice 

The  author  believes  that  strategic  leaders  should  eschew  the 
common  approach  of  attempting  to  develop  programs  and  policies 
to  motivate  the  workforce,  or  at  least  any  workforce  similar  to  this 
population.  Leaders  cannot  force  motivation.  There  is  no  causal  rela¬ 
tionship.  Leaders  can  mold  an  environment  that  allows  workers  to 
motivate  themselves,  but  typical  attempts  to  extrinsically  motivate 
workers  are  counterproductive.  The  approach  or  philosophy  of  the 
leadership,  as  Sprenger  (2007)  suggested,  should  be  to  concentrate 
their  time  and  resources  on  identifying  and  developing  programs 
and  policies  that  eliminate  the  negative  aspects  of  workforce  per¬ 
formance.  There  appears  to  be  a  greater  return  on  investment  for 
this  approach. 

Implications  for  Leaders 

The  supervisor-subordinate  relationship,  as  Buckingham  and 
Coffman  (1999)  suggest,  appears  to  be  a  critical  — possibly  the 
most  critical  — relationship  in  the  workplace.  A  poor  supervisor- 
subordinate  relationship  is  the  leading  cause  of  employee  attrition 
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(MSPB,  2010,  p.  2).  On  the  positive  side,  the  “relationship  with 
managers”  ranked  as  the  sixth  most  important;  on  the  negative 
side,  “bad  managers”  ranked  second.  While  a  few  respondents 
were  concerned  with  anonymity  of  answers,  many  more  reportedly 
shared  their  responses  with  supervisors  during  annual  appraisal 
feedback  sessions.  Supervisors  should  consider  a  similar  discussion 
with  all  subordinates.  The  exchange  of  perceptions  may  improve  the 
supervisor-subordinate  relationship. 

Implications  for  Human  Resources  Organizations 

If  agencies  insist  upon  individual  performance  evaluations— a 
position  not  supported  by  the  author— agencies  should  consider 
implementing  a  360-degree  performance  evaluation  system  instead 
of  the  traditional  supervisor-only  system.  While  a  recent  poll  of 
federal  workers  by  Federal  Times  (“How  Should  the  Job  Perfor¬ 
mance...,”  2010)  indicates  that  only  19  percent  of  federal  workers 
believe  these  evaluations  to  be  effective  (likely  as  a  method  to 
identify  a  “bad  manager”),  a  large  body  of  anecdotal  evidence 
suggests  that  360-degree  feedback  systems  are  effective  in,  at  a 
minimum,  bringing  poor  supervisory  performance  to  the  attention 
of  senior  management.  The  single  viewpoint  of  a  biased,  inattentive, 
or  English-challenged  supervisor  may,  in  some  cases,  paint  a  false 
picture  of  employee  performance. 

However,  be  aware  that  in  industry,  the  general  acceptance  of 
360-degree  evaluations  is  diminishing.  The  common  belief  appears 
to  be  that  these  evaluations  are  useful,  but  very  expensive;  and  the 
measured  increases  in  performance  and  profitability  cannot  justify 
these  costs.  While  federal  agencies  are  cost-constrained,  they  are 
not  constrained  by  the  profit  motive.  Such  360-degree  evaluations 
may  be  an  appropriate  cost  of  doing  business,  especially  in  organi¬ 
zations  where  the  public  perception  of  high  performance  is  critical. 


Final  Thoughts 

The  acquisition  community  is  dominated  by  knowledge  work¬ 
ers.  These  highly  educated,  high-skilled  workers  are  self-managed 
and  self-motivated.  The  traditional  management  approaches  that 
appeared  effective  for  the  assembly-line  workers  of  yesteryear  are 
counterproductive  when  applied  to  the  knowledge-based  work¬ 
force.  The  monumental  challenge  for  today’s  leaders  is  to  abandon 
the  management  practices  of  the  last  50  years,  which  to  some  are 
counterintuitive  and  fraught  with  uncertainty,  and  to  embrace  a 
theory  that  is  still  emerging.  Extrinsic-based  attempts  to  “motivate 
the  workforce,”  despite  conventional  wisdom,  are  ineffective.  Pay 
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for  performance,  bonuses,  and  even  traditional  performance  evalu¬ 
ation  systems,  in  the  opinion  of  this  author,  are  anachronisms. 

When  Thomas  Paine  said,  “lead,  follow,  or  get  out  of  the  way,”  he 
did  not  have  the  acquisition  community  in  mind,  but  his  admonish¬ 
ment  is  appropriate  for  today’s  leaders.  The  recipe  for  “doing  more 
without  more”  is  a  simple  one— one  part  solid,  insightful  leadership 
and  two  parts  “getting  out  of  the  way.” 
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REQUIREMENTS  AND 
COST  STABILITY: 

A  CASE  STUDY  OF  THE 
F/A-18  HORNET  PROGRAM 

f  CDR  Jay  D.  Bottelson,  USN 

Most  government  and  industry  leaders  involved  with  Depart¬ 
ment  of  Defense  acquisition  programs  emphasize  the 
importance  of  requirements  and  cost  stability.  However, 
despite  all  the  stated  support  for  program  element  stability 
and  acquisition  reform,  frequent  changes  are  experienced  in 
acquisition  programs  that  affect  the  final  end  product  in  terms 
of  changes  to  unit  design,  number  of  units  procured,  system 
and  subsystem  capability,  as  well  as  affecting  the  overall  cost 
of  the  program.  This  study  analyzes  the  U.S.  Navy’s  F/A-18A 
model  to  identify  requirements  changes;  discern  the  reasons 
for  change  and  the  impact  the  resultant  change  made  on 
the  program  (funding,  schedule,  capacity,  etc.);  and  develop 
recommendations  for  limiting  requirements  creep,  instability, 
and  cost  growth  in  future  programs. 


Keywords:  Acquisition  Reform,  Cost  Growth, 
Requirements  Stability,  Requirements  Creep, 
F/A-18  Hornet 
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"Among  the  changes  made  in  the  acquisition 

PROCESS  IN  THE  LAST  20  YEARS  HAVE  BEEN  THE  GREATLY 
INCREASED  EMPHASIS  ON  PROGRAM  MANAGEMENT, 
WITH  CAPITAL  LETTERS.  IT  COULD  BE  NOTED  THAT 
THERE  SEEMS  TO  BE  A  FAIR  DEGREE  OF  CORRELATION 
BETWEEN  THAT  GROWTH  IN  EMPHASIS  WITH 
SEVERITY  OF  THE  ACQUISITION  PROBLEM  IN  TERMS 
OF  LENGTHENED  SCHEDULES  AND  INCREASED  COSTS.” 
(SPANGENBERG,  1981) 

—George  Spangenberg 

<>*x»o<><><x><><><x><><><><><><><><><>c«><x><><x><x><x><x>o<><>^^ 

Ironically,  the  most  successful  modern  U.S.  Navy  aircraft  began 
its  life  as  a  U.S.  Air  Force  prototype.  Therein  lies  the  lineage  of  the 
F/A-18  Flornet.  Indeed,  the  F/A-18  evolved  from  what  can  only  be 
termed  as  a  bizarre  set  of  circumstances,  tracing  its  beginnings  back 
to  the  1960s  when  the  Air  Force  began  looking  for  a  lightweight 
fighter  (Jenkins,  2000).  By  the  mid-1970s,  the  Navy  and  Air  Force 
were  directed  to  work  together  and  field  a  common  lightweight 
fighter.  Following  a  fly-off  between  the  final  two  competing  pro¬ 
totype  aircraft,  the  Air  Force  chose  its  champion,  which  ultimately 
became  the  F-16  Fighting  Falcon.  At  the  time  and  inexplicably,  the 
Navy  demurred  and  chose  the  loser  of  the  competition. 

This  article  examines  the  topics  of  acquisition  reform,  require¬ 
ments  stability,  and  cost  growth  to  determine  the  forces  behind 
changes  in  major  acquisition  programs  and  what  drives  the 
changes— threats,  technology,  schedule,  budget,  or  performance. 
While  acquisition  reform  presently  gets  plenty  of  headlines,  it  has 
been  an  issue  in  the  defense  arena  for  years,  as  highlighted  by 
the  ongoing  annual  assessments  of  defense  weapon  programs  by 
the  Government  Accountability  Office  (GAO,  2009).  Yet  despite 
the  stated  desire  for  requirements  stability,  frequent  changes  are 
experienced  in  acquisition  programs  that  affect  the  final  end  prod¬ 
uct  in  terms  of  significant  changes  to  unit  design,  number  of  units 
procured,  system  and  subsystem  capability,  and  unit  costs  to  name 
but  a  few  variables. 

This  study  scrutinizes  the  initial  fielded  version  of  the  Flornet, 
the  F/A-18A,  as  a  basis  for  study  of  the  acquisition  process  and 
the  requirements  and  capabilities  changes  that  occurred  between 
program  approval  and  final  product  fielding.  It  will  investigate  why 
the  F/A-18  was  needed  and  the  timeline  for  development,  what  the 
initial  program  requirements  and  cost  estimates  were,  and  what 
changes  and  adjustments  were  made.  In  examining  these  changes 
and  adjustments,  it  delves  into  the  causes  and  effects,  namely  why 
changes  were  necessary  and  what  were  the  costs  of  the  changes. 
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Finally,  this  article  attempts  to  analyze  and  suggest  a  means  for 
improving  future  program  performance  by  identifying  these  past 
concerns.  Specifically,  it  will  attempt  to  discover  the  reason  and 
amount  of  change  from  the  initial  plan  in  terms  of  time,  cost,  or 
product  performance  and  capability.  Additionally,  it  will  categorize 
the  impact  of  the  changes  on  the  program  and  develop  lessons 
learned  and  recommendations  for  limiting  requirements  creep, 
instability,  and  cost  growth. 


Preparing  for  Launch 

For  a  number  of  years,  the  Navy  moved  along  toward  filling  its 
fleet  of  fighter  aircraft  with  the  highly  capable  F-14  Tomcat.  But  in 
1971,  the  deputy  secretary  of  defense,  following  the  recommenda¬ 
tions  set  forth  in  the  Five  Year  Defense  Plan,  limited  the  Navy  to 
only  313  F-14A  fighters  (F-18, 1975).  At  about  the  same  time,  the  Air 
Force  opened  competition  for  design  of  a  lightweight  fighter.  In 
mid-1973,  the  Department  of  Defense  (DoD)  and  Congress  placed 
strong  pressure  on  the  Navy  for  significant  cost  increases  occur¬ 
ring  in  the  F-14  fighter  program.  Additionally,  Congress  felt  that  the 
Navy  should  pursue  a  lightweight  fighter  as  well,  and  the  secretary 
of  defense  directed  the  Navy  to  assess  the  Air  Force  lightweight 
fighter  designs  (Jenkins,  2000).  By  the  spring  of  1974,  two  proto¬ 
types  were  ready  for  test  flights— the  General  Dynamics  YF-16  and 
the  Northrop  YF-17  (Kelly,  1990). 

The  Navy  and  Air  Force  both  ultimately  battled  back-and-forth 
with  the  DoD  over  what  they  felt  their  needs  were,  and  what  DoD 
wanted  them  to  have.  The  Air  Force  abruptly  changed  course  and 
attempted  to  make  the  lightweight  fighter  effort  go  away  by  under- 
funding  it,  while  a  Navy  fighter  study  group  recommended  several 
variants  of  the  F-14  without  the  expensive  Phoenix  air-to-air  missile 
(Stevenson,  1993).  Despite  its  efforts,  the  Air  Force  was  thwarted 
when  the  Office  of  the  Secretary  of  Defense  (OSD)  decided  to  pro¬ 
cure  the  YF-16  lightweight  fighter  for  the  Air  Force.  This  was  done 
by  inserting  funding  in  the  Fiscal  Year  (FY)  1975  budget  request 
sent  to  Congress  in  January  1974—1  month  before  the  YF-16  con¬ 
ducted  its  first  test  flight  (Stevenson,  1993). 

In  spite  of  direction  from  both  Congress  and  DoD,  the  Navy 
released  a  Presolicitation  Notice  (PSN)  to  industry  for  its  own 
lightweight  fighter— the  VFAX  (V-fixed  wing,  F-fighter,  A-attack, 
and  X-experimental).  Plowever,  Congress  turned  it  down  in  August 
1974  and  placed  it  under  a  new  program  name  called  the  NACF,  or 
Navy  Air  Combat  Fighter  (F-18,  1975).  As  the  Air  Force  continued 
toward  acquisition  of  its  new  fighter,  in  September  1974  the  Joint 


391 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


392 


Committee  on  Appropriations  weighed  in  on  this  issue.  The  commit¬ 
tee  directed  that  the  Navy  would  make  appropriate  modifications 
to  the  winner  of  the  air  combat  fighter  competition  (Stevenson, 
1993).  This  process  to  achieve  commonality  between  the  Services 
for  their  lightweight  fighter  needs  made  as  much  sense  then  as  it 
does  today  because  it  would  reduce  overhead  and  simplify  support 
issues.  Unfortunately,  as  the  late  Senator  Barry  Goldwater  observed 
at  the  time,  “the  only  way.. .to  get  the  Navy  and  Air  Force  to  agree  on 
a  common  fighter  aircraft,  is  to. ..lock  Navy  and  Air  Force  designers 
in  the  same  room  until  they  could  agree...”  (F-18,  1975). 

When  the  YF-16  was  announced  as  the  winner  of  the  Air  Force 
lightweight  fighter  competition  in  January  1975,  the  Navy  was  not 
happy.  According  to  Gaddis  (2003),  too  much  modification  was 
required  to  “naval-ize”  the  aircraft,  such  as  widening  the  distance 
between  the  rear  landing  gear,  adding  a  keel,  strengthening  the 
airframe  and  all  landing  gear,  and  installing  a  tailhook— all  to  accom¬ 
modate  catapulting  and  arrested  landings  on  aircraft  carriers  at  sea. 
This  would  essentially  result  in  a  new  aircraft  that  would  definitely 
not  have  the  commonality  that  Congress  and  others  desired,  and 
would  weigh  considerably  more  as  well.  Consequently,  the  Navy 
requested  and  received  approval  to  develop  the  YF-17,  the  loser  of 
the  Air  Force  fly-off  competition,  and  “[i]n  a  rare  bout  of  bureau¬ 
cratic  honesty,  ...redesignated  the  aircraft  F-18  in  recognition  of  the 
substantial  differences”  (Jenkins,  2000). 


"IT  IS  MY  OPINION  THAT  THE  AIR  FORCE  WITH  THE  F-16 
AND  THE  NAVY  WITH  THE  F-18  FIND  THEMSELVES  TODAY 
IN  THE  POSITION  OF  DEVELOPING  AN  AIRCRAFT  FOR 
WHICH  NEITHER  HAD  AN  ORIGINAL  REQUIREMENT. 

This  doesn’t  mean  each  service  cannot  use  these 
aircraft...  Fortunately,  the  services  have  great 

FLEXIBILITY  WHICH  ENABLES  THEM  TO  SURVIVE  OUR 
COLLECTIVE,  BUT  SOMETIMES  NOT  TOO  WISE,  POLITICAL 
WISDOM." 

—Senator  Barry  Goldwater 
Statement  to  the  Senate  Appropriations 
Committee,  October  21, 1975 


Initial  Vector-Requirements  and  Cost 

When  it  came  time  for  contracts  to  be  written  for  the  develop¬ 
ment  of  the  F-18,  the  Navy  program  manager  called  for  something 
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unique  at  the  time.  Newly  hired  from  the  National  Aeronautics 
and  Space  Administration,  he  called  for  specifications  written  into 
the  contract  for  reliability  in  addition  to  performance.  Previously, 
reliability  had  been  addressed  in  contracts  as  goals,  but  never  as 
specifications.  When  finalized,  the  contracts  indeed  captured  the 
first  ever  agreement  by  a  contractor  to  deliver  reliability,  maintain¬ 
ability,  and  performance  (Kelly,  1990). 

Those  first  contracts  were  issued  in  January  1976  for  develop¬ 
ment  and  production  for  the  first  11  planes.  However,  there  were 
plans  to  deliver  three  versions  of  the  aircraft— an  F-18  fighter  and 
A-18  light  attack  aircraft  for  the  Navy,  and  dual-purpose  aircraft  for 
the  Marine  Corps  that  was  very  close  to  the  F/A-18  that  was  finally 
fielded  (Kelly,  1990).  Initially,  780  aircraft  were  planned  to  go  to 
the  Navy  and  the  Marine  Corps.  Some  exceptional  engineering 
and  development  of  a  dual-use  radar  for  both  air-to-air  and  air-to- 
ground  use  allowed  the  F-18  and  A-18  designs  to  merge.  As  a  result, 
the  Hornet  began  to  be  called  the  F/A-18  in  1980,  and  was  fielded 
in  two  versions:  the  single-seat  F/A-18A  and  the  two-seat  F/A-18B 
(Jenkins,  2000).  Initial  operational  capability  (IOC)  was  scheduled 
for  1983  (Dyer,  1981,  p.  13). 

As  previously  stated,  this  article  examines  the  F/A-18A  model, 
which  was  developed,  produced,  and  delivered  from  FY1975  to 
FY1985.  During  this  timeframe,  371  total  F/A-18A  aircraft  were  deliv¬ 
ered  before  the  changeover  to  production  of  the  next  model— the 
F/A-18C.  Additionally,  41  of  the  F/A-18B  versions  were  delivered 
during  the  same  period.  Foreign  military  sales  of  F/A-18A  and  B 
versions  were  also  produced  and  sold  to  Australia  (52  aircraft), 
Canada  (115  aircraft),  and  Spain  (30  aircraft)  during  the  same  period 
(Jenkins,  2000). 

The  requirements,  or  performance  standards,  for  the  F-18  were 
initially  described  in  the  PSN  of  June  1974.  The  PSN  described  the 
initial,  or  threshold,  requirements  as  well  as  the  final,  or  goal,  require¬ 
ments.  (Goal  requirements  are  currently  referred  to  as  objective 
requirements.)  Though  all  requirements  are  important,  ultimately 
some  can  tend  to  be  more  important  than  others.  However,  several 
requirements  proved  difficult  to  attain  during  development  and 
flight  testing,  such  as  operating  range  (specifically  how  far  the 
aircraft  could  fly  on  internal  fuel),  acceleration,  and  overall  aircraft 
weight  (General  Accounting  Office,  1980a). 

The  threshold  operating  radius  for  the  F-18  was  400  nautical 
miles  (NM),  with  a  goal  of  550  NM.  It  was  to  be  able  to  accelerate 
from  0.8  Mach  to  1.6  Mach  in  120  seconds  at  35,000  feet  threshold, 
and  80  seconds  goal.  Finally,  it  was  to  have  gross  takeoff  weight  of 
30,000  pounds  or  less  (Stevenson,  1993).  These  three  performance 
requirements  were  not  the  only  ones  to  cause  problems,  but  they 
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will  be  the  main  focus  within  this  article  due  to  their  significance  for 
fighter  aircraft.  In  addition  to  performance  concerns,  cost  growth 
caused  just  as  much  apprehension  then  as  it  does  today. 

When  the  Defense  Systems  Acquisition  Review  Council  approved 
full-scale  development  of  the  F-18  in  December  1975,  the  desired 
flyaway  design-to-cost  goal  was  $5.6  million  in  FY1975  dollars  (Coo¬ 
per,  1978).  The  first  quarterly  reports  for  the  F-18,  titled  Selected 
Acquisition  Reports,  or  SARS,  began  shortly  thereafter.  SARs  were 
transmitted  to  Congress  to  report  on  the  progress  and  cost  esti¬ 
mates  of  DoD  major  acquisition  programs.  The  first  report  on  the 
F-18  (Office  of  the  Under  Secretary  of  Defense,  1976)  stated  that: 

The  initial  F-18  SAR... provides  for  a  program  of  11  R&D  and  800 
production  aircraft  at  an  overall  cost  of  $12,831.1M,  comprised  of 
$8,005.6M  in  FY  1975  constant  dollars  and  $4,825.5M  in  escala¬ 
tion,  based  on  an  average  annual  rate  of  5.2%.  This  equates  to 
a  FY  1975  constant  dollar  program  unit  cost  of  S9.871M  and  an 
escalated  unit  cost  of  $15.821M.  (p.  2) 

Costs  for  major  aircraft  acquisition  programs  can  be  classified 
in  three  ways:  flyaway  cost,  procurement  cost,  or  program  cost. 
These  costs  are  depicted  in  the  Figure.  Flyaway  cost  includes  the 
basic  airframe,  the  engine,  avionics,  self-contained  armament,  and 
any  equipment  furnished  by  the  government  to  the  contractor  for 
inclusion  in  the  aircraft.  Procurement  cost  takes  flyaway  cost  and 
adds  support  and  training  equipment,  technical  data  and  publica¬ 
tions,  technical  services  provided  by  the  contractor,  and  initial  spare 
parts  required.  Program  cost  then  takes  procurement  cost  and  fur¬ 
ther  adds  research  and  development  and  any  military  construction 
costs  to  reach  a  total  acquisition  cost. 

FIGURE.  ACQUISITION  PROGRAM  COSTS 


Airframe  Engine  Avionics  Armament 

Flyaway  Cost 

Support  Spares  Services 

Procurement  Cost 

RDT&E  MILCON 

Program  Cost 

A  problem  that  exists  with  expressing  cost  in  three  different 
ways  is  that  it  can  get  very  confusing  to  those  charged  with  over¬ 
sight  of  the  complete  program.  For  example,  the  cost  that  seems 


Requirements  and  Cost  Stability: 

A  Case  Study  of  the  F/A-18  Hornet  Program 


October  2011 


most  often  described  in  congressional  testimony  reviewed  for  this 
study  was  flyaway  cost.  While  not  truly  a  misrepresentation,  flyaway 
cost  does  not  tell  the  complete  story.  As  the  table  shows,  flyaway 
cost  can  play  down  the  true  cost  of  the  program  effort,  sometimes 
by  as  much  as  50  percent. 


TABLE.  MAJOR  AIRCRAFT  ACQUISITION  PROGRAMS-THREE  MAIN 
COST  CLASSIFICATIONS 


FY77  Budget 

FY78  Budget 

FY79  Budget 

FY75  $  M 

TY  $  M 

FY75  $  M 

TY  $  M 

FY75  $  M 

TY  $  M 

Flyaway 

6.14 

10.33 

6.13 

10.21 

6.33 

11.8 

Procurement 

8.19 

13.71 

8.15 

13.52 

8.33 

15.3 

Program 

9.87 

15.82 

9.95 

15.8 

10.16 

17.6 

Note.  Adapted  from  Congressional  Research  Service,  Report  No.  78-224-F,  December  15,  1978. 
TY  =  Then  year;  $  M  =  Dollars  (in  millions) 


As  the  SAR  excerpt  previously  described,  costs  are  mainly 
expressed  as  a  combination  of  program  costs  and  escalation  costs. 
Program  cost  variance  can  be  due  to  changes  in  quantity,  changes 
in  requirements  or  capabilities,  inflationary  or  deflationary  cost 
changes,  contractor  overhead  rates,  delivery  date  changes,  or  even 
foreign  military  sales  (General  Accounting  Office,  1981).  Current 
programs  experience  change  mainly  due  to  increased  research, 
development,  test,  and  evaluation  costs,  program  growth  costs, 
delay  in  delivery  of  initial  capabilities,  and  decreases  in  planned 
quantities  (Government  Accountability  Office,  2009). 


Mid-Course  Corrections— Causes  and  Effects 

The  F/A-18A  program  experienced  cost  growth  relatively  early 
in  the  development  phase  as  well  as  throughout  its  production  run 
until  FY1985,  when  block  changes  were  incorporated  to  upgrade  the 
F/A-18A  and  B  versions  to  C  and  D  versions  (Elward,  2000).  How¬ 
ever,  early  program  growth  was  mainly  due  to  a  mismatch  between 
the  inflation  rates  the  program  office  was  required  to  use  by  OSD. 
Guidance  from  the  Office  of  Management  and  Budget  (OMB) 
directed  OSD  to  use  inflation  rates  from  the  economic  assumptions 
contained  in  the  President’s  Budget  (General  Accounting  Office, 
1981).  For  example,  the  March  1976  SAR  listed  a  5.2  percent  rate, 
but  the  General  Accounting  Office  found  in  1980  that  OSD  infla¬ 
tion  rates  ranged  from  5.4  percent  to  6.3  percent.  Yet  for  the  same 
time  period,  the  Bureau  of  Labor  Statistics  calculated  a  13  percent 
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inflation  estimate,  and  the  Air  Force  derived  a  19  percent  aerospace 
industry  inflation  rate  (General  Accounting  Office,  1980b). 

This  required  use  of  low  inflation  rates  had  two  detrimental 
effects.  It  made  the  Service  appear  to  be  understating  program 
costs,  and  it  made  budgeting  difficult.  In  one  case,  a  Navy  official 
stated  that  the  FY1981  budget  submission  would  have  been  15  per¬ 
cent  higher  if  the  aerospace  industry  inflation  rate  had  been  used 
(General  Accounting  Office,  1981).  This  correlates  to  an  explanation 
in  the  notes  for  the  December  1980  SAR  (Office  of  the  Under  Sec¬ 
retary  of  Defense,  1980). 

Program  costs  increased  by  $8,177.9M,  from  $29,712.3M  to 
$37,890.2M,  due  to  (1)  the  application  of  higher  anticipated 
escalation  rates  in  program  outyears  ($+451. 0M),  (2)  procure¬ 
ment  schedule  stretchout  ($+907.0M),  (3)  support  increase  as  a 
result  of  revised  basing  plan  and  repricing  of  support  program 
($+728. 7M),  and  (4)  reestimate  of  the  R&D  and  procurement 
programs  ($+6,091.2M).  Of  the  $6,091.2M  estimating  increase, 
$3,855.8M  represents  the  difference  between  Government 
inflation  projections  and  actual  experience  on  the  FY  1979- 
1981  production  contracts  and  proposals.  As  a  consequence, 
support  purchases  have  been  deferred  and  seven  aircraft  are 
being  dropped  from  the  FY  1981  buy.  Both  actions  contribute 
significantly  to  the  schedule  and  support  purchases  identified 
above,  (p.  2) 

An  evaluation  of  the  numbers  just  cited  clearly  reflects  that  63 
percent  of  the  cost  for  the  re-estimate  of  the  R&D  and  procurement 
programs  was  due  to  the  difference  in  inflation  rate  projections  and 
actual  costs  experienced  during  previous  years. 

Another  program  cost,  though  not  monetary,  occurred  in  the 
form  of  reduced  performance  capabilities.  During  the  flight  test 
phase  of  development,  there  were  demonstrated  shortfalls  in  a  num¬ 
ber  of  key  areas.  The  acceleration  threshold  of  120  seconds  from  0.8 
to  1.6  Mach  described  in  the  PSN  was  lowered  to  110  seconds  in  the 
contract  specification.  At  the  first  flight  evaluation  in  March  1979,  it 
took  156  seconds.  By  May  1980,  the  contractor  achieved  the  accel¬ 
eration  in  116  seconds,  but  could  not  reach  the  target  specification 
(Stevenson,  1993).  Additionally,  range  thresholds  were  not  met  dur¬ 
ing  the  demonstration  phase.  Despite  an  operating  radius  threshold 
of  400  NM  on  internal  fuel,  the  best  range  eventually  achieved  was 
380  NM  after  significant  work  by  the  contractor  and  testing  by  both 
the  contractor  and  the  Navy  (Jenkins,  2000). 

In  1981,  a  report  on  the  F/A-18  explained  that  OSD  (and  thus  the 
Navy  as  well)  had  decided  that  “the  demonstrated  acceleration  and 


396 


Requirements  and  Cost  Stability: 

A  Case  Study  of  the  F/A-18  Hornet  Program 


October  2011 


range  were  acceptable,”  despite  being  well  short  of  threshold  speci¬ 
fications  (General  Accounting  Office,  1981).  However,  one  cause 
for  concern  here  in  retrospect  is  why  substandard  performance  of 
program  thresholds  and  contract  specifications  was  accepted.  The 
perception  given  is  that  the  F/A-18  was  wanted  at  any  cost,  even 
with  reduced  performance.  Quite  simply,  it  was  an  instance  of  speci¬ 
fications  not  being  met  and  then  changed,  or  de-scoped,  because 
they  could  not  be  met. 

The  F/A-18  was  also  nearly  2,000  pounds  over  its  initial  speci¬ 
fication  weight  according  to  a  General  Accounting  Office  (1980) 
report.  Some  of  this  weight  growth  was  due  to  combining  the 
designs  for  the  F-18  and  A-18,  as  the  attack  variant  was  144  pounds 
heavier  than  the  fighter  version.  Nearly  500  pounds  was  added 
due  to  reliability  and  maintainability  features,  and  another  1,300 
pounds  for  engineering  estimates  to  attain  reliability  and  main¬ 
tainability  goals  (General  Accounting  Office,  1980a).  These  goals 
were  added  after  the  PSN  was  issued  as  part  of  the  new  program 
manager’s  attempt  to  write  reliability  into  the  contract  as  well  as 
performance.  Although  the  specification  called  for  a  gross  takeoff 
weight  of  33,652  pounds,  the  weight  of  the  aircraft  demonstrated 
during  evaluation  was  35,363  pounds.  Eventually,  the  Navy  changed 
the  weight  specification  because  it  became  nearly  36,000  pounds 
(Stevenson,  1993).  This  is  an  example  of  requirements  creep  on  an 
upward  scope.  So  once  again,  a  specification  was  changed  to  meet 
a  design  shortcoming. 


Final  Destination— Hitting  the 
Requirements  Target 

Though  the  F/A-18A  Hornet  proved  its  worth  in  war  and  peace, 
it  was  a  very  different  aircraft  from  what  was  initially  envisioned, 
designed,  and  estimated  for  cost.  Indeed,  when  evaluated  in  terms 
of  schedule,  cost,  and  performance,  the  F/A-18A  only  attained 
one  of  the  three  criteria  to  effective  standards.  The  Hornet  was 
produced  with  minimal  slippage  in  terms  of  development  and 
production  timeline,  delivering  the  first  production  model  in  April 
1980,  and  the  first  aircraft  to  IOC  in  January  1983  (Boeing,  n.d.). 
This  compares  extremely  well  with  present  day  major  programs, 
where  only  28  percent  manage  to  achieve  IOC  on  time  (Government 
Accountability  Office,  2009). 

Cost  was  an  area  where  better  execution  should  have  been 
attained.  The  F/A-18A  began  as  a  $12.8  billion  program  ($8  billion 
for  the  base  program  and  $4.8  billion  for  projected  escalation  costs) 
in  FY1975.  Ten  years  later,  in  FY1985,  it  had  grown  immensely  to 
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become  a  $39.3  billion  program.  This  was  partly  due  to  an  additional 
buy  of  566  aircraft  that  added  $6.8  billion  in  base  year  dollars  and 
$19.7  billion  in  actual  and  projected  escalation  costs,  but  that  por¬ 
tion  of  the  cost  growth  is  not  really  a  major  concern  in  this  case. 
However,  the  choice  to  purchase  more  aircraft,  while  it  does  raise 
the  cost  of  the  program,  is  certainly  not  indicative  of  program  mis¬ 
management,  and  can  often  lower  the  unit  cost  through  economy 
of  scale. 

Escalation  costs  were  the  single  largest  factor  for  cost  increases 
in  the  F/A-18A  program,  an  observation  made  in  nearly  every  Gen¬ 
eral  Accounting  Office  report  on  the  Hornet.  While  the  December 
1985  SAR  shows  the  percent  of  cost  growth  attributed  to  total 
adjustment  for  quantity  as  99  percent,  a  quick  calculation  shows 
the  growth  of  Actual  and  Projected  Escalation  under  Current  Esti¬ 
mate-Program  Cost  was  over  400  percent  (Office  of  the  Under 
Secretary  of  Defense,  1985).  Though  it  may  appear  that  the  Navy 
was  grossly  deficient  in  their  budgetary  management  for  allowing 
this  cost  growth,  the  culpability  lies  with  the  required  use  of  OMB 
inflation  rates  that  were  lower  than  real  inflation. 

Lastly,  while  the  F/A-18A  became  a  top-performing  fighter/ 
attack  aircraft,  the  Navy  made  tradeoffs  during  development  and 
accepted  less  performance  than  was  originally  specified  in  both  the 
PSN  and  the  contract.  The  Navy  approved  reductions  to  the  con¬ 
tract  specifications  of  9.4  percent  for  range  and  11.2  percent  for  level 
flight  acceleration  (General  Accounting  Office,  1980a).  Additionally, 
it  was  forced  to  change  the  weight  specifications  when  engineering 
and  design,  as  well  as  the  requirements  for  reliability  and  maintain¬ 
ability,  caused  the  weight  of  the  aircraft  to  grow  beyond  contract 
specifications. 


Post-Flight  Debrief— Finding  and  Fixing  Gripes 

On  the  whole,  the  F/A-18A  program  had  several  successes.  First, 
it  was  produced  without  major  adjustments  to  the  development  and 
delivery  schedule.  In  fact,  it  went  from  contractor  selection  to  the 
first-delivered  production  model  in  just  under  5  years.  Second,  it 
featured  the  first  instance  where  a  contractor  was  tasked  to  deliver 
reliability,  maintainability,  and  performance  as  part  of  the  contract. 
This  was  a  fair  achievement  for  the  F/A-18  in  general.  Though  reli¬ 
ability  and  maintainability  exceeded  expectations  and  significantly 
reduced  life-cycle  costs,  it  also  caused  the  overall  system  weight  to 
increase.  Finally,  it  achieved  all  this  while  undergoing  a  number  of 
significant  engineering  changes,  such  as  merging  the  F-18  and  A-18 
models  into  a  single  aircraft  and  developing  a  next  type  of  radar. 
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Thus,  from  a  program  management  perspective,  the  F/A-18A  was 
exceptionally  well-managed. 

Conversely,  the  study  that  formed  the  basis  of  analysis  for  this 
article  identified  two  key  problems  experienced  between  concept 
and  fielding  of  the  final  product  that  are  causes  for  concern.  The 
first  issue  was  cost  growth  due  to  “uncontrollable  factors,”  as 
the  General  Accounting  Office  (1980b)  report  called  them.  As 
discussed  previously,  escalation  cost  increases  driven  by  inflation 
rates  were  a  key  factor  in  overall  cost  growth.  Yet  the  inflation 
rates  used  by  the  Navy  were  stipulated  by  OSD,  as  directed  by 
OMB  (General  Accounting  Office,  1980b).  This  issue  was  well  docu¬ 
mented  in  General  Accounting  Office  reports  from  1980  to  1998, 
and  even  for  the  latest  version  of  the  Hornet— the  F/A-18E/F.  The 


1998  report  stated  that  OSD-directed  rates  were  still  lower  than 
the  industry  averages  (General  Accounting  Office,  1998).  Further 
review  of  more  recent  OSD  guidance  from  February  2005  found 
guidance  to  use  a  2.0  percent  inflation  rate  for  FY2006  (Office  of 
the  Under  Secretary  of  Defense,  2005),  while  the  Bureau  of  Labor 
Statistics  shows  aerospace  industry  inflation  rates  of  4.8  percent  for 
aircraft  manufacturing  for  the  same  period  (U.S.  Bureau  of  Labor 
Statistics,  2006). 

Reduced  performance  capabilities  are  the  second  key  issue. 
The  F/A-18A  was  not  able  to  meet  several  specified  performance 
requirements  during  testing  and  demonstration.  The  choice  pre¬ 
sented  was  to  either  require  the  contractor  to  deliver  the  aircraft  as 
specified— running  a  risk  of  cost  and  schedule  overruns— or  accept 
an  aircraft  with  reduced  capabilities.  The  Navy  chose  to  accept  the 
reduction  in  capabilities,  and  this  was  not  a  new  instance  of  doing 
so.  Historically,  the  Navy  had  similar  problems  with  the  F-14  Tomcat. 
A  comparison  of  requirements  to  fielded  capabilities  revealed  that 
the  F-14  was  5,000  pounds  overweight,  failed  to  meet  required 
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ranges,  could  not  attain  combat  ceiling,  and  missed  required  main¬ 
tenance  and  reliability  as  well  as  several  other  requirements.  The 
F-14  did  not  meet  its  cost  target  either  (F-18, 1975). 

Accordingly,  these  two  key  issues  can  be  considered  as  major 
“gripes”  of  enduring  significance  that  need  to  be  fixed.  In  aviation 
terms,  a  gripe  is  a  maintenance  problem  that  must  be  repaired 
before  the  aircraft  can  fly  again.  For  the  first  gripe,  as  shown  previ¬ 
ously,  OSD  is  using  OMB  inflation  rates  that  are  statistically  too  low. 
These  inflation  rates  resulted  in  two  substantial  problems  for  the 
Navy.  It  made  the  Navy  appear  to  be  minimizing  program  costs,  and 
thus  caused  the  Navy  to  be  suspect  in  the  eyes  of  those  charged 
with  program  oversight.  It  also  caused  problems  for  those  respon¬ 
sible  for  preparing  budgets,  especially  when  they  knew  through  past 
experience  that  the  inflation  rates  would  not  meet  real  economic 
averages.  At  a  minimum,  Bureau  of  Labor  Statistics  inflation  rates 
should  be  used.  Ideally,  Air  Force  aerospace  industry  rates  should 
be  used. 

The  second  gripe  focuses  on  requirements  instability.  It  was 
expected  that  this  study  would  show  requirements  being  added 
after  program  start— a  phenomenon  referred  to  as  requirements 
creep  that  is  common  in  present-day  programs.  This  was  not  the 
case.  Instead,  the  instability  was  in  the  Navy  and  DoD  holding 
firm  on  the  specifications  given  to  the  contractor  for  delivery.  As 
described  previously,  the  F/A-18A  came  up  short  in  a  number  of 
performance  capabilities.  For  future  programs,  the  Navy  (or  any 
Service  for  that  matter)  and  DoD  should  decide  before  program 
start  whether  to  accept  performance  standard  shortfalls,  and  if 
so,  how  much  variance  is  acceptable.  This  can  be  done  by  setting 
threshold  and  final  performance  goals  that  focus  on  attainment  of 
a  short  range  of  parameters  in  the  contract.  For  example,  instead 
of  specifying  a  top  performance  speed  of  1.8  Mach,  a  range  of  1.6  to 
1.8  Mach  is  specified,  with  the  lower  number  considered  minimally 
acceptable  and  the  higher  number  desired.  To  encourage  the  con¬ 
tractor  to  reach  the  higher  standard,  a  scaled  award  fee  or  incentive 
fee  could  be  used  to  reward  the  contractor  for  achievements  above 
the  minimum  requirement.  Alternatively,  if  there  is  not  a  desire  to 
accept  reduced  performance,  there  must  be  consideration  to  how 
much  additional  time  or  cost  growth,  or  even  both,  is  acceptable  to 
reach  the  desired  performance  standards. 
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Concluding  Thoughts  and  Recommendations 

The  F/A-18A  Hornet  was  born  of  a  process  that  started  with 
clean  sheets  of  paper  and  a  need  to  field  a  lightweight  fighter  to 
complement  the  F-14  Tomcat.  Along  the  way,  it  went  up  against  the 
highest  levels  of  DoD  and  Congress,  danced  around  the  Air  Force 
lightweight  fighter  competition,  and  emerged  as  a  truly  exceptional 
fighter/attack  aircraft.  The  fact  that  it  managed  to  stay  on  schedule 
and  achieve  IOC  in  light  of  the  myriad  engineering  changes  required 
to  merge  the  F-18  and  A-18  models  is  a  testament  to  the  dedication 
of  the  designers,  engineers,  and  program  manager. 

However,  in  spite  of  these  dedicated  efforts,  the  program  dealt 
with  several  major  obstacles  in  the  form  of  escalation/inflation 
costs  and  the  inability  of  the  aircraft  to  meet  required  specifica¬ 
tions.  Though  the  escalation  costs  did  not  detract  from  the  ability 
to  see  the  program  through  to  the  end,  they  gave  the  perception 
of  a  program  that  was  not  being  managed  properly  and  was  being 
deliberately  understated  in  order  to  continue  to  receive  funding. 
Additionally,  though  the  performance  shortcomings  did  not  stop 
the  F/A-18A  from  achieving  IOC  and  success  in  the  fleet,  concern 
remains  over  the  reasoning  behind  accepting  less  than  what  was 
called  for  in  the  contract  specifications. 

To  ensure  the  Services  and  taxpayers  get  the  most  for  their 
money,  the  DoD  needs  to  make  two  major  changes:  (a)  revise 
the  way  it  calculates  and  allows  for  inflation  in  major  acquisition 
programs,  and  (b)  base  such  calculations  on  more  realistic  values, 
such  as  those  provided  by  the  Bureau  of  Labor  Statistics.  It  must 
also  reevaluate  and  enforce  established  processes  for  situations 
where  programs  cannot  meet  specified  requirements.  This  includes 
options  of  whether  to  agree  to  a  lower  performance  level  or  to  push 
for  the  established  requirement  to  be  met,  with  the  acknowledg¬ 
ment  and  acceptance  of  the  fact  that  it  could  take  more  time,  incur 
a  higher  cost,  or  both.  The  words  of  George  Spangenberg  at  the 
beginning  of  this  article  are  as  true  today  as  they  were  when  writ¬ 
ten  in  1981.  One  additional  quote  of  his  bears  repeating  and  brings 
this  conclusion  to  a  proper  closing:  “We  should  return  to  optimiz¬ 
ing  the  naval  aircraft  acquisition  process,  rather  than  accepting 
compromise  in  the  name  of  federal  procurement  standardization” 
(Spangenberg,  1981). 
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BETTER  SCHEDULE 
PERFORMANCE 
ASSESSMENTS 
DERIVED  FROM 
INTEGRATED  MASTER 
PLAN-REFERENCED 
SCHEDULE  METRICS 


\  David  C.  Bachman 

The  integrated  master  plan  (IMP)  provides  a  better  structure 
than  either  the  work  breakdown  structure  (WBS)  or  organi¬ 
zational  breakdown  structure  for  measuring  actual  integrated 
master  schedule  (IMS)  progress.  The  author  posits  that 
improved  understanding  of  schedule  performance  and  better 
identification  of  program  risks  result  when  an  IMP  structure 
is  evaluated  in  addition  to  the  earned  value  management- 
mandated  IMS  WBS  structure.  The  article  examines  how 
the  “Hit-Miss”  index,  baseline  execution  index,  and  critical 
path  length  index  (CPLI)  were  used  to  evaluate  the  life-cycle 
performance  of  a  12-month,  900-task  IMP  program  event. 
CPLI,  the  author  concludes,  is  subject  to  interpretation  and 
must  be  evaluated  against  four  caveats:  duration  remaining, 
total  float  including  schedule  margin,  schedule  compression, 
and  schedule  avoidance. 


Keywords:  Baseline  Execution  index  (BEI), 

Critical  Path  Length  index  (CPU),  Earned  Value 
Management  (EVM),  Integrated  Master  Plan  (IMP), 

Integrated  Master  Schedule  (IMS) 
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Since  2005,  the  National  Geospatial-Intelligence  Agency  (NGA) 
has  been  tasked  under  the  Base  Realignment  and  Closure  Act  to 
consolidate  all  Washington,  DC,  metropolitan  area  facilities  to  a 
standalone  campus  currently  called  NGA  Campus  East  (NCE)  by 
September  11,  2011.  The  U.S.  Army  Corps  of  Engineers  is  currently 
managing  the  construction  of  NCE  at  the  Fort  Belvoir,  Virginia, 
Engineering  Proving  Grounds  (NGA,  2010).  In  addition  to  the  NCE 
facilities,  NGA  has  awarded  additional  contracts  for  the  installation 
of  communication,  hardware,  and  software  systems  necessary  to 
support  the  NGA  mission  at  the  new  facility.  This  article  examines 
the  schedule  performance  reported  from  one  of  these  contracts. 
The  contract  included  full  earned  value  management  (EVM)  imple¬ 
mentation,  and  the  reported  data  came  from  an  Electronic  Industries 
Alliance  (EIA)-748B-compliant  earned  value  management  system 
(EVMS)  (Government  Electronics  and  Information  Technology 
Association  [GEIA],  2007). 

The  NGA  program  management  team  has  included  EVM  as 
one  tool  to  effectively  manage  program  risk,  technical  scope,  cost, 
and  schedule.  This  contract’s  effort  is  built  around  an  integrated 
master  plan  (IMP)1  consisting  of  11  major  program  events  (PE),2  43 
significant  accomplishments,3  and  201  accomplishment  criteria.4 
From  this  IMP,  a  6,000-line  integrated  master  schedule  (IMS)  has 
evolved  and  continues  to  grow  each  month  as  the  contract  matures. 
A  product-oriented  work  breakdown  structure  (WBS)  and  a  corre¬ 
sponding  EVM  performance  measurement  baseline  (PMB)  resulted 
from  the  IMS.  Integrated  baseline  reviews  (IBR)  were  held  in  month 
2  and  month  6  of  the  program,  and  all  IBR-related  issues  were 
resolved  by  month  13.  The  NGA  EVM  Center  of  Excellence  (EVM 
COE)  is  responsible  for  program  oversight  and  was  challenged  to 
create  a  set  of  pure,  “straightforward”  IMS  metrics  unrelated  to  EVM 
that  would  provide  NGA  leadership  with  accurate  assessments  of 
schedule  progress. 

The  EVM  COE  augmented  the  contract-level  acquisition,  tech¬ 
nology  and  logistics  (AT&L)  tripwire  schedule  metrics  to  improve 
their  utility  for  assessing  NCE  contract  schedule  progress.  Initially, 
the  EVM  COE  examined  reporting  on  all  14  of  the  Defense  Contract 
Management  Agency  (DCMA)’s  14  point  schedule  assessment5 
metrics  (Treacy,  2010)  using  the  proposed  62-element  Generally 
Accepted  Scheduling  Principle  (GASP),6  a  quick-look  schedule 
assessment  (Meyer,  2010).  Both  proved  to  be  far  too  detailed  and 
intricate  to  address  the  straightforward  challenge  from  NGA  lead¬ 
ership.  In  2006,  DCMA  standardized  a  set  of  EVM  and  schedule 
metrics  for  the  Defense  Acquisition  Executive  Summary  (DAES) 
process  known  as  the  AT&L  tripwire  metrics.  A  subset  of  DCMA’s  14 
point  schedule  assessment  metrics  is  included  as  AT&L  tripwire  met- 
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rics.  The  two  primary  and  seven  secondary  metrics  are  designed  to 
surface  problems  early  for  effective  issue  resolution  (Kester,  2007). 
The  baseline  execution  index  (BEI)  (which  measures  work  progress) 
and  the  critical  path  length  index  (CPLI)  (which  measures  efficiency 
associated  with  completing  a  milestone)  are  two  of  the  secondary 
AT&L  tripwire  metrics  directly  related  to  schedule  performance. 
Although  not  directly  reported  as  a  tripwire  metric,  the  DCMA  BEI 
tripwire  briefing  also  reports  the  closely  related  “Task  Hit/Miss 
Percentage”  or  the  “Hit-Miss”  index.  These  three  metrics  were  the 
starting  point  for  NGA’s  straightforward  schedule  assessment.  Ini¬ 
tially,  the  EVM  COE  computed  these  metrics  only  at  the  contract 
level  and  found  them  to  have  limited  utility.  Department  of  Defense 
(DoD)  policy  and  the  IMS  data  item  description  require  the  IMS  to 
be  delivered  in  a  product-oriented  WBS  format  (DoD,  2006).  Since 
most  of  the  contract’s  PEs  cut  across  multiple  WBS  elements,  com¬ 
puting  WBS-related  BEIs  and  Hit-Miss  indexes  revealed  little  about 
progress  to  the  next  PE.  Considering  that  September  2011  was  many 
months  away,  the  critical  path  to  that  date  was  mostly  controlled 
by  EVM  summary-level  planning  packages7  or  external  milestones, 
making  a  contract-level  CPLI  at  best  misleading  and  unreliable.  To 
accurately  assess  contract  progress,  the  EVM  COE  computed  IMS 
schedule  metrics  using  an  IMP  structure  and  redefined  the  CPLI 
tripwire  metric  to  include  schedule  margin.8 

After  reviewing  an  actual  12-month  IMP  PE  life  cycle,  five  metrics 
emerged  that  best  defined  schedule  performance  and  status:  con- 
tract-level  Hit-Miss  index,  PE  Hit-Miss  index,  contract-level  BEI,  PE 
BEI,  and  the  PE  CPLI.  The  EVM  product-oriented  WBS  provided 
little  insight  into  actual  schedule  performance  because  each  report¬ 
ing-level  WBS  element  supported  multiple  PEs.  The  IMS  data  item 
description  requires  the  IMS  to  be  vertically  traceable  to  the  IMP, 
but  it  includes  the  caveat  “(if  applicable)”  (DoD,  2005b).  NGA  con¬ 
tractors  are  required  to  map  their  IMS  tasks  and  milestones  to  the 
IMP.  This  allows  the  schedule  to  be  sorted  by  IMP  PEs,  IMP  signifi¬ 
cant  accomplishments,  as  well  as  the  EVM  product-oriented  WBS 
and  contract  organization  structures.  Because  the  contract  PEs 
were  sequential  in  nature,  the  program  management  team’s  assess¬ 
ment  focused  on  the  next  IMP  PE  or,  in  one  case,  the  next  two  PEs 
because  they  were  being  completed  in  parallel.  The  EVM  COE  Hit- 
Miss  index  uses  the  AT&L  Hit-Miss  equation  to  measure  the 
percentage  of  the  current  month  baseline  tasks/activities  actually 
completed  (or  Hit)  on  or  ahead  of  their  baseline  schedule  (Figure 
1).  The  EVM  COE  uses  the  AT&L  tripwire  BEI  equation  to  measure 
the  cumulative  efficiency  with  which  actual  work  is  accomplished 
when  measured  against  the  baseline  (Figure  2).  Different  from  the 
AT&L  tripwire  equation,  the  NGA  CPLI  equation  includes  recognition 


October  2011 


407 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


408 


FIGURE  1.  “HIT-MISS”  INDEX  EQUATION 


Task 

#  of  THIS  month's  tasks  finished  on 

or  ahead  of  their  baseline  schedule 

Hit -Miss 

#  of  THIS  month's  tasks  to  be 

finished  in  the  baseline  schedule 

Note.  This  current  period  metric  measures  the  percentage  of  current  month  baseline  tasks/ 
activities  actually  completed  (or  Hit)  on  or  ahead  of  their  baseline  schedule  (Hurley,  2007). 


FIGURE  2.  BASELINE  EXECUTION  INDEX  (BEI)  EQUATION 


BEI 


#  of  Baseline  Tasks  Actually  Completed 
#  of  Baseline  Tasks  Scheduled  to  be  Completed 


Note.  This  cumulative  metric  measures  the  efficiency  with  which  actual  work  has  been 
accomplished  when  measured  against  the  baseline  (Hurley,  2007). 


of  IMS  schedule  margin.9  By  dissecting  the  CPLI  over  the  entire  life 
cycle  of  an  IMP  PE,  the  program  management  team  discovered  that 
unlike  the  Hit-Miss  and  BEI  metrics,  they  could  easily  influence  the 
value  of  the  CPLI  metric  (Figure  3).  To  truly  understand  a  CPLI,  four 
schedule  caveats  must  be  determined:  duration  remaining,  float10 
and  margin,  schedule  compression,11  and  schedule  avoidance.12  The 
following  discussion  presents  a  hypothetical  schedule  to  review  and 
explain  these  metrics  and  CPLI  schedule  caveats. 

FIGURE  3.  CRITICAL  PATH  LENGTH  INDEX  (CPLI)  EQUATION 

Remaining  Duration  +  Float  +  Schedule  Margin 

CPLI  =  - 

Remaining  Duration 


Note.  Indexes  the  remaining  duration  to  an  IMP  PE’s  current  finish  (or  to  the  original 
baseline  finish,  whichever  is  greater)  plus  float  duration  plus  schedule  margin  against  the 
remaining  duration.  Float  duration  is  always  measured  to  the  IMP  PE's  baseline  finish  date. 


Method 

The  NGA  schedule  metrics  and  rationale  for  the  four  CPLI  cave¬ 
ats  can  best  be  explained  by  reviewing  the  progress  of  a  hypothetical 
schedule.  Figure  4  is  a  hypothetical  baseline  schedule  consisting  of 
summary-level  tasks,  work  tasks,  one  margin  task,  two  PE  mile¬ 
stones,  and  a  start  milestone.  The  network13  schedule  to  PE  No.  1 
includes  a  critical  path  string  and  a  high-risk  path  string,  each  net¬ 
worked  to  the  PE  No.  l’s  multistring  margin  task.  Remember  that 


FIGURE  4.  BASELINE  SCHEDULE 
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TABLE  1.  SCHEDULE  METRICS  FOR  HYPOTHETICAL  SCHEDULE 


Month 

II 

Hit-Miss"a 

BEIb 

CPLIC 

of 

2010 

PT 

HT 

Index 

CT 

FT 

Index 

Index 

DR 

F+M 

SC 

SA 

January 

2 

1 

0.50 

2 

2 

1.00 

|  1.15 

65 

10 

0 

0 

February 

3 

2 

0.67 

5 

5 

1.00 

|  1.16 

45 

7 

0 

0 

March 

3 

1 

0.33 

8 

6 

0.75 

|  1.13 

22 

3 

2 

0 

April 

3 

2 

0.67 

11 

9 

0.81 

|  1.80 

5 

4 

0 

9 

May 

2 

13 

June 

1 

14 

Note.  “Hit-Miss”  =  "Hit-Miss”  Index;  BEI  =  Baseline  Execution  Index;  CPU  =  Critical  Path  Length  Index;  PT  =  No. 
Period  Tasks;  HT  =  No.  Hit  Tasks;  CT  =  No.  Cumulative  Tasks;  FT  =  No.  Finished  Tasks;  DR  =  Duration  Remaining  in 
Days;  F+M  =  Total  Float  +  Margin  in  Days;  SC  =  Schedule  Compression  in  Days;  SA  =  Schedule  Avoidance  in  Days. 
a”Hit-Miss”:  Green  >  0.75;  Gray  >  0.25  and  <  0.75;  Orange  <  0.25.  bBEI:  Green  >  0.90;  Gray  >  0.75  and  < 
0.90;  Orange  <  0.75.  CCPLI:  Green  >1.05;  Gray  >  1.00  and  <  1.05;  Orange  <  1.00. 


the  critical  path  is  the  longest  path  through  a  network  schedule  and 
may  not  represent  the  high-risk  path.  The  high-risk  string  in  this 
schedule  has  5  days  of  total  float.  To  demonstrate  the  NGA  metrics 
and  the  CPLI  caveats  observed  on  the  NGA  contract,  all  tasks  asso¬ 
ciated  with  the  critical  path  will  be  completed  on  schedule,  and  the 
high-risk  tasks  will  finish  late  to  demonstrate  how  the  metrics 
change  as  the  schedule  slips.  These  metrics  and  their  color  coding 
are  summarized  in  Table  1. 

NGA  Schedule  Metrics 

Hit-Miss  Index,  BEI,  and  CPLI.  The  January  29,  2010,  schedule 
(Figure  5)  and  the  January  2010  data  (Table  1)  highlight  the  basic 
calculations  associated  with  the  NGA  schedule  metrics.  At  the  end 
of  the  first  reporting  period  (January  29,  2010),  two  tasks  were 
scheduled  to  be  completed  and  both  were  completed.  Since  Task 
A1  finished  on  schedule,  it  counts  as  a  “Hit”  for  the  Hit-Miss  index 
and  as  a  completed  task  for  the  BEI.  Task  B2  finished  3  workdays 
late,  but  still  during  the  reporting  period  so  it  counts  as  a  “Miss”  for 
the  Hit-Miss  and  as  a  completed  task  for  the  BEI.  The  Hit-Miss  index 
is  1  a  2  =  0.50,  and  the  BEI  is  2  a  2  =  1.00.  Although  the  high-risk 
path  slipped,  it  has  not  yet  slipped  onto  the  critical  path.  So  the  IMS 
critical  path  to  PE  No.  1  is  still  controlled  by  the  critical  path  string 
(0  float  days)  and  the  margin  task  (10  days).  The  January  29,  2010, 
schedule  task— duration  remaining  check— shows  that  65  workdays 
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FIGURE  5.  JANUARY  29,  2010,  SCHEDULE:  “HIT-MISS”  INDEX,  BEI,  &  CPLI 
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remain  until  the  PE  No.  1  baseline  finish  date,  resulting  in  a  1.15  CPLI 
[(65  +  0  +  10)  -r  65  =  1.15]  for  the  first  reporting  period. 

CPLI  Schedule  Caveats 

CPLI— duration  remaining  and  total  float.  The  February  26,  2010, 
schedule  (Figure  6)  indicates  the  high-risk  path  tasks  have  redefined 
the  schedule’s  critical  path.  The  February  2010  line  entry  in  Table 
1  lists  the  February  26,  2010,  metrics.  At  the  end  of  the  second 
reporting  period,  all  five  tasks  scheduled  to  be  completed  have  been 
completed,  resulting  in  a  1.00  BEI.  Remember  that  the  Hit-Miss  index 
is  a  current  period  metric  so  the  0.67  value  represents  Task  A2  and 
Task  A3  finishing  on  schedule  (“Hit”)  and  Task  B2  finishing  8  days 
late.  With  45  days’  duration  remaining,  the  schedule’s  critical  path 
is  now  defined  by  the  slipping  high-risk  path  tasks.  To  maintain  the 
April  30,  2010,  PE  No.  1  baseline  finish  date,  the  margin  task  was 
reduced  from  10  days  to  7  days,  resulting  in  a  1.16  CPLI  [(45  +  0  + 
7)  -r  45  =  1.16].  Despite  the  fact  that  the  high-risk  path  continues  to 
slip,  the  CPLI  showed  a  slight  improvement,  thereby  demonstrating 
the  need  for  the  duration  remaining  caveat  and  total  float  caveat. 

CPLI— schedule  compression.  The  March  26,  2010,  schedule 
(Figure  7)  introduces  schedule  compression  and  an  unfavorable 
BEI.  The  March  2010  line  entry  in  Table  1  lists  the  March  26,  2010, 
metrics.  The  high-risk  path  tasks  slipped  an  additional  6  days  in 
March,  and  only  Task  A4  finished  on  schedule.  Neither  Task  B3  nor 
Task  B4  was  completed,  resulting  in  the  Table  1  March  2010  Hit- 
Miss  index  of  0.33  and  BEI  of  0.75.  If  the  schedule  margin  task’s 
duration  was  reduced  for  the  entire  March  26,  2010,  schedule  slip 
(6  days),  a  yellow  1.045  CPLI  would  result.  By  compressing  the  Task 
B5’s  schedule  by  2  days,  the  CPLI  stays  green  because  the  float  + 
margin  remain  at  plus  3  days.  This  raises  an  obvious  question:  Can 
Task  B5  be  completed  in  8  versus  10  days?  It  also  highlights  why 
schedule  compression  is  a  dimension  that  must  be  considered  when 
evaluating  the  CPLI  metric. 

CPLI— schedule  avoidance  The  April  23,  2010,  schedule  (Figure  8) 
documents  the  final  CPLI  schedule  caveat  observed  on  the  contract. 
The  April  2010  line  entry  in  Table  1  lists  the  April  23,  2010,  metrics. 
Schedule  avoidance  occurs  when  a  task  is  eliminated  from  the 
schedule,  the  IMS  network  logic  is  changed  to  allow  a  task  to  be 
deferred  to  a  later  program  event  milestone,  or  the  IMS  logic  is 
changed  to  allow  sequential  tasks  to  be  done  in  parallel.  During 
April  2010,  the  overall  schedule  slipped  an  additional  7  days,  making 
an  on-time  delivery  of  PE  No.  1  impossible.  To  support  an  on-time 


FIGURE  6.  FEBRUARY  26,  2010,  SCHEDULE:  CPLI-DURATION  REMAINING  AND  TOTAL  FLOAT 
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FIGURE  7.  MARCH  26,  2010,  SCHEDULE:  CPLI-SCHEDULE  COMPRESSION 
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FIGURE  8.  APRIL  23,  2010:  CPLI-SCHEDULE  AVOIDANCE 
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delivery  of  PE  No.  1,  completion  of  Task  B5  is  deferred  to  PE  No.  2. 
This  is  done  by  deleting  Task  B5’s  task  relationship  with  the  schedule 
margin  task  and  linking  it  only  to  the  PE  No.  2  milestone.  Task  B5’s 
logic  now  bypasses  PE  No.  1,  resulting  in  the  avoidance  of  9  days  of 
duration  remaining,  1  additional  day  of  duration  for  the  margin  task, 
and  a  very  favorable  1.80  CPLI  [(5  +  0  +  4)  -f  5  =  1.80]  for  PE  No.  1. 
Without  the  Task  B5  schedule  avoidance,  the  PE  No.  1  milestone 
slips  4  days,  and  the  CPLI  would  be  0.56  =  [{9  +  (-4)  +  0}/  9].  This 
highlights  why  schedule  avoidance  is  a  CPLI  metric  caveat  that  must 
be  evaluated. 


Results  and  Discussion 

The  metrics  and  CPLI  caveats  discussed  hypothetically  in  this 
article  are  computed  and  documented  for  the  actual  program  in 
Tables  2,  3,  and  4  at  both  contract  and  IMP  PE  levels.  Tables  2,  3, 
and  4  document  the  contract  schedule  metrics  for  IMP  PE-E,  PE-F, 
and  PE-G,  respectively.  The  data  reflect  month  24  for  PE-E,  and 
month  25  for  PE-F  and  PE-G.  PE-G  is  the  immediate  successor 
event  to  PE-E  and  is  being  completed  in  parallel  with  PE-F,  which 
is  unrelated.  Both  PE-G  and  PE-F  are  baselined  to  finish  during  the 
first  2  weeks  of  month  35.  The  contract-level  data  tasks  only  include 
tasks  through  day  15  of  month  35. 

Many  conclusions  can  be  drawn  from  the  data  contained  in 
Tables  2,  3,  &  4  and  the  IMP  strategy  used  to  collect  them.  Con¬ 
clusions  associated  with  Table  2  are  historic  in  nature  and  were 
confirmed  with  the  NGA  program  manager  and  contractor.  It  is 
important  to  note  that  PE-E  was  completed  on  time.  This  was 
achieved  by  deferring  some  PE-E  tasks  to  PE-G,  and  by  eliminating 
other  PE-E  tasks  that  were  not  required.  In  the  author’s  opinion, 
identification  of  the  deferred  and  eliminated  tasks  may  not  have 
been  discovered  using  traditional  EVM  WBS  or  Organizational 
Breakdown  Structure  (OBS)  analysis  strategies.  Applying  what  the 
program  management  team  learned  from  PE-E,  pertinent  ques¬ 
tions  arise  regarding  PE-F  and  PE-G  trends  and  overall  schedule 
performance.  Seven  key  conclusions  supported  by  this  contract’s 
data,  metrics,  and  the  IMP  analysis  strategy  follow: 

1.  Unfavorable  schedule  trends.  In  all  cases,  unfavorable  sched¬ 
ule  trends  are  first  manifested  in  the  Flit-Miss  index,  then  the 
BEI,  and  lastly  in  the  CPLI. 

2.  Identification  of  work  scope  changes.  IMP  strategy  clearly 
identifies  work  scope  changes.  The  total  number  of  PE-E 
tasks  gradually  increased  from  1,109  tasks  in  month  14  to  1,171 
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FT  =  No.  Finished  Tasks;  DR  =  Duration  Remaining  in  Days;  F+M  =  Total  Float  +  Margin  in  Days;  SC  =  Schedule  Compression  in  Days;  SA  =  Schedule  Avoidance  in  Days. 
aData  based  on  end  of  Month  24  schedule.  b"Hit-Miss":  Green  >  0.75;  Gray  >  0.25  and  <  0.75;  Orange  <  0.25.  CBEI:  Green  >  0.90;  Gray  >  0.75  and  <  0.90;  Orange  <  0.75. 
dCPLI:  Green  >  1.05;  Gray  >  1.00  and  <  1.05;  Orange  <  1.00. 


TABLE  3.  SCHEDULE  METRICS  FOR  PROGRAM  EVENT  F 
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Note.  "Hit-Miss"  =  "Hit-Miss"  Index;  BEI  =  Baseline  Execution  Index;  CPLI  =  Critical  Path  Length  Index;  PT  =  No.  Period  Tasks;  HT  =  No.  Hit  Tasks;  CT  =  No.  Cumulative  Tasks; 

FT  =  No.  Finished  Tasks;  DR  =  Duration  Remaining  in  Days;  F+M  =  Total  Float  +  Margin  in  Days;  SC  =  Schedule  Compression  in  Days;  SA  =  Schedule  Avoidance  in  Days. 

“Data  based  on  end  of  Month  25  schedule.  Inconsistencies  between  Table  2  &  Table  3  Contract  Level  metrics  reflect  incomplete  tasks  deleted  from  baseline  during  month 
25.  b"Hit-Miss":  Green  >  0.75;  Gray  >  0.25  and  <  0.75;  Orange  <  0.25.  CBEI:  Green  >  0.90;  Gray  >  0.75  and  <  0.90;  Orange  <  0.75.  dCPLI:  Green  >  1.05;  Gray  >  1.00  and  < 
1.05;  Orange  <  1.00. 
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tasks  in  month  21.  The  program  management  team  con¬ 
firmed  that  these  changes  resulted  from  the  conversion  of 
EVM  planning  packages  to  EVM  work  tasks,  and  to  the  addi¬ 
tion  of  new  work  resulting  from  contract  changes.  In  month 
22,  the  total  tasks  associated  with  PE-E  were  abruptly 
reduced  to  924  tasks.  The  IMP  strategy  analysis  approach 
clearly  identified  the  descoping,  and  it  was  confirmed  that 
the  257  associated  tasks  were  either  deferred  or  eliminated 
to  support  PE-E’s  month  24  delivery  date  (Figure  9).  Addi¬ 
tionally,  comparing  the  Table  2  (based  on  month  24  data) 
contract  cumulative  tasks  for  months  22-24  with  the  same 
months  on  Tables  3  and  4  (based  on  month  25  data),  the 
program  management  team  can  account  for  the  permanent 
elimination  of  90  PE-E  tasks  from  the  schedule. 


FIGURE  9.  PE-E  WORK  SCOPE  CHANGES 


3.  IMP  synergistic  planning  focus.  The  IMP  structure  allows  for 
the  current  month  contract-level  tasks  to  be  associated  with 
the  current  IMP  program  focus.  Tables  3  and  4  and  Figure 
10  show  a  total  of  99  contract  tasks  baselined  for  month  25; 
83  percent  of  these  tasks  are  associated  with  either  PE-F  (22 
tasks)  or  PE-G  (60  tasks).  This  leaves  17  current  month  tasks 
that  are  not  associated  with  either  of  the  next  two  PEs.  In 
and  of  themselves,  the  WBS  and  OBS  structures  would  not 
easily  support  this  type  of  assessment. 

4.  IMP  synergistic  performance  focus.  Closely  related  to  con¬ 
clusion  3,  comparing  the  current  month  “Contract  Tasks 
Hit”  and  the  corresponding  PE-F  and  PE-G  “Hits”  task  val¬ 
ues  provides  insight  on  the  contract-level  metric.  Figure  10 
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FIGURE  10.  IMP  SYNERGISTIC  METRICS 


Month  25  Month  25 

Baseline  Tasks  Scheduled  Baseline  Tasks  Completed 

shows  30  of  the  99-month,  25  contract-level  schedule  tasks 
were  completed.  Subtracting  the  7  Hits  from  PE-F  and  the  7 
Hits  from  PE-G  leaves  16.  So  16  of  the  17  tasks  not  associated 
with  either  of  the  next  two  milestones  were  completed.  This 
suggests  the  contract-level  Hit-Miss  index  by  itself  may  not 
be  a  good  indicator  for  assessing  PE-F  or  PE-G  progress. 

5.  BEI  measures  schedule  performance  trends.  Evaluation 
of  the  PE  BEI  trend  can  indicate  if  a  schedule  is  improv¬ 
ing.  Review  of  the  PE-E  (Table  2)  BEI  metric  from  month 
16  to  month  18  indicates  the  number  of  cumulative  tasks 
increased  at  a  rate  faster  than  the  finished  tasks  were  being 
completed.  The  month  16  BEI  of  0.57  deteriorated  to  0.48 
by  month  18,  indicating  a  slipping  schedule.  This  fact  is  also 
supported  by  the  total  float  value,  which  eroded  by  8  days 
during  the  same  period.  Like  the  EVM  schedule  performance 
index  metric,  the  BEI  eventually  must  improve  to  1.00.  This 
is  clearly  shown  in  the  BEI  improving  from  0.48  in  month  18 
to  0.98  in  month  24. 

6.  CPLI  schedule  compression  an  “early”  tripwire.  Schedule 
compression  was  first  observed  after  the  PE-E  CPLI  metric 
went  from  green  to  yellow.  In  month  18,  PE-E’s  CPLI  went 
from  1.06  (an  NGA  green  value)  to  1.03  (an  NGA  yellow 
value).  In  the  following  month’s  schedule,  the  metric  returned 
to  green  (1.11)  when  the  baseline  durations  of  three  future 
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critical  path  tasks  were  reduced  by  a  total  of  17  days.  In 
many  instances,  the  schedule  compression  magnitude  was 
greater  on  the  near-critical  path  strings  than  on  the  critical 
path  string.  The  author  attributes  this  to  control  account 
managers  wanting  to  avoid  the  extra  scrutiny  associated 
with  being  on  the  critical  path.  The  EVM  COE  makes  no 
value  judgment  on  the  validity  of  the  schedule  compression; 
as  such,  the  NGA  CPLI  equation  (Figure  3)  does  not  con¬ 
sider  it.  Schedule  compression  is  a  key  caveat  the  program 
manager  must  consider  when  evaluating  schedule  risk.  The 
author  believes  schedule  compression  is  a  critical  indicator 
of  pending  schedule  issues.  In  the  PE-E  schedule,  schedule 
compression  was  first  observed  in  month  19—3  months 
before  the  month  22  adjustment  discussed  in  conclusion  2. 

7.  CPLI  schedule  avoidance  “too  late”  tripwire.  Schedule  avoid¬ 
ance  first  appeared  when  it  became  apparent  that  PE-E, 
as  it  was  originally  baselined,  could  not  be  completed  on 
schedule.  During  the  final  30  workdays  leading  up  to  PE-E, 
39  tasks  originally  associated  with  PE-E  were  remapped 
to  PE-F.  This  allowed  for  the  on-time  delivery  of  PE-E.  The 
impact  of  this  can  be  observed  in  the  month  23  “PT”  col¬ 
umn  (Table  4),  which  denotes  PE-G  Tasks  Baselined.  The 
contractor  confirmed  that  the  disproportionate  number  of 
month  23  tasks  (152)  resulted  from  tasks  being  transferred 
from  PE-E.  Like  schedule  compression,  the  EVM  COE  does 
not  consider  schedule  avoidance  in  the  CPLI  computa¬ 
tion,  but  does  report  it  as  a  CPLI  schedule  metric  caveat. 
The  program  manager  must  make  the  final  decision  on  the 
potential  cost,  schedule,  and  programmatic  risk  associated 
with  eliminating  work  scope  altogether  or  deferring  work 
scope  to  a  later  IMP  PE  milestone. 
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Summary  and  Concluding  Discussion 

The  EVM  COE  was  challenged  to  create  a  set  of  schedule  met¬ 
rics  that  provide  NGA  leadership  with  accurate  assessments  of 
schedule  progress.  A  better  understanding  of  schedule  perfor¬ 
mance  and  improved  program  risk  identification  were  realized  on 
this  contract  when  NGA  focused  its  schedule  metrics  on  the  IMP 
structure.  While  this  study’s  positive  results  are  based  on  a  single 
contract,  they  justify  additional  research  with  a  larger  data  set.  The 
CPLI  caveats  qualify  the  CPLI  tripwire  metric,  which  could  be  easily 
misinterpreted  without  them.  Considering  this  finding,  additional 
research  is  warranted  to  justify  requiring  CPLI  metric  reports  to 
include  the  caveats.  NGA  program  managers  have  embraced  the 
IMP  structure  and  the  metrics  discussed  in  this  article  because  they 
are  straightforward,  focus  on  near-term  problems,  and  identify  spe¬ 
cific  tasks  needed  to  assess  programmatic  and  schedule  risk.  This 
IMP  approach,  however,  does  not  address  cost.  The  EVM  COE  found 
that  the  IMP  structures  were  so  different  from  the  contract’s  cor¬ 
responding  WBS  and  OBS  structures  that  EVM  cost  and  schedule 
data  could  only  be  correlated  at  the  total  contract  level. 

NGA  has  applied  the  techniques  discussed  in  this  article  with 
mixed  success  on  other  NGA  contracts.  When  the  IMP  and  sched¬ 
ule  margin  were  well-defined,  the  resulting  schedule  metrics  were 
easily  computed  and  meaningful.  However,  often  the  required  IMP 
was  poorly  constructed  and  schedule  margins  were  ill-defined. 
Schedule  compression  and  schedule  avoidance,  to  a  lesser  extent, 
were  observed  on  many  NGA  contracts.  The  438-page  Government 
Accountability  Office  (GAO)  Cost  Estimating  Assessment  Guide 
lists  only  three  IMP  references  and  contains  no  discussion  on  how 
IMP  is  to  be  used  (GAO,  2009).  The  EVM  EIA-748B  standard  does 
not  reference  the  IMP  at  all  (GEIA,  2007).  For  the  past  6  years,  the 
National  Defense  Industrial  Association  (NDIA,  n.d.)  has  identified 
schedule  margin  as  an  unresolved  issue  (Treacy,  2009;  Berkey 
2004).  The  community  needs  to  create  an  IMP  data  item  descrip¬ 
tion  and  establish  a  best  practice  for  implementing  schedule  margin 
to  universally  realize  meaningful,  straightforward  schedule  metrics 
based  on  this  article’s  IMP  approach. 

The  IMP  structure  augmented  and  provided  more  meaningful 
metrics  for  measuring  near-term  schedule  performance  then  either 
the  WBS  or  OBS  structures;  however,  integrated  program  manage¬ 
ment  requires  cost,  schedule,  and  performance  metrics.  This  study 
highlights  the  value  of  adding  an  IMP  structure  to  the  IMS.  The  logi¬ 
cal  extension  of  this  study  would  be  to  also  require  EVM  data  be 
mapped  to  the  IMP  structure. 
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ENDNOTES 

1.  Integrated  Master  Plan  (IMP)— The  IMP  is  an  event-based  plan  consisting  of  a 
hierarchy  of  program  events,  with  each  event  being  supported  by  specific  significant 
accomplishments,  and  each  accomplishment  is  associated  with  specific  accomplishment 
criteria  to  be  satisfied  for  its  completion  (DoD,  2005a). 

2.  Program  Event— A  program  event  is  a  program  assessment  point  that  occurs  at 
the  culmination  of  significant  program  activities:  significant  accomplishments  and 
accomplishment  criteria  (DoD,  2005a). 

3.  Significant  Accomplishment— A  significant  accomplishment  is  the  desired  result(s)  prior 
to  or  at  completion  of  a  program  event  that  indicates  a  level  of  the  program’s  progress 
(DoD,  2005a). 

4.  Accomplishment  Criteria— Accomplishment  criteria  provide  definitive  evidence  that  a 
specific  significant  accomplishment  has  been  completed  (DoD,  2005a). 

5.  Defense  Contract  Management  Agency  14  Point  Schedule  Assessment— A  set  of 
standardized  schedule  heath  and  performance  metrics  used  to  evaluate  integrated 
master  schedules.  The  metrics  included:  logic,  leads,  lags,  task  relationships,  constraints, 
high  float,  negative  float,  high  duration,  invalid  dates,  resources,  missed  tasks,  critical 
path,  CPLI,  and  BEI. 

6.  Generally  Accepted  Scheduling  Principle  (GASP)— A  defense  industry-Department 
of  Defense  initiative  to  produce  valid  and  effective  schedules.  To  meet  GASP  tenets, 
a  schedule  must  be  complete,  traceable,  transparent,  statused,  predictive,  usable, 
resourced,  and  controlled. 

7.  Summary-Level  Planning  Package  (SLPP)— An  aggregation  of  work  for  far-term  efforts, 
not  comprised  of  detailed  planning  nor  able  to  be  identified  at  the  control  account  level, 
which  can  be  assigned  to  reporting-level  WBS  elements  (DoD,  2006). 

8.  Schedule  Margin— A  management  method  for  accommodating  schedule  contingencies. 

It  is  a  designated  buffer  and  shall  be  identified  separately  and  considered  part  of  the 
baseline.  Schedule  margin  is  the  difference  between  contractual  milestone  date(s)  and 
the  contractor’s  planned  date(s)  of  accomplishment  (DoD,  2005b). 

9.  IMS  Schedule  Margin— There  are  differing  opinions  in  the  EVM  community  on  the  proper 
use  and  interpretation  of  IMS  schedule  margin8  (DCMA,  2010;  Price,  2008;  NDIA,  n.d.). 
Because  most  NGA  contracts  include  schedule  margin  strategies,  the  EVM  COE  metrics 
include  the  use  of  schedule  margin. 

10.  Float— Also  known  as  total  float  and  total  slack.  The  amount  of  time  a  task/activity  or 
milestone  can  slip  before  it  delays  the  contract  or  project  finish  date  (DoD,  2005b). 

11.  Schedule  Compression— Schedule  Compression  is  the  difference  between  a  task’s 
baseline  duration  and  the  task’s  current  duration.  Microsoft  Project  calls  this  duration 
variance  (Stover,  2007). 

12.  Schedule  Avoidance— Schedule  Avoidance  occurs  when  a  task’s  baseline  logic  is 
changed  to  bypass  a  measured  milestone. 

13.  Network— A  schedule  format  in  which  the  activities  and  milestones  are  represented  along 
with  the  interdependencies  between  work  tasks  and  planning  packages  (or  lower  level 
tasks  or  activities).  It  expresses  the  logic  (i.e.,  predecessors  and  successors)  of  how  the 
program  will  be  accomplished.  Network  schedules  are  the  basis  for  critical  path  analysis, 
a  method  for  identification  and  assessment  of  schedule  priorities  and  impacts.  At  a 
minimum,  all  discrete  work  shall  be  included  in  the  network  (DoD,  2005a). 
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APPENDIX 

List  of  Abbreviations  and  Acronyms 

AT&L  Acquisition,  Technology  and  Logistics 

BEI  Baseline  Execution  Index 

CPLI  Critical  Path  Length  Index 

DAES  Defense  Acquisition  Executive  Summary 

DCMA  Defense  Contract  Management  Agency 

DoD  Department  of  Defense 

EIA  Electronic  Industries  Alliance 

EVM  Earned  Value  Management 

EVM  COE  NGA  Earned  Value  Management  Center  of  Excellence 
GAO  Government  Accountability  Office 

GASP  Generally  Accepted  Scheduling  Principle 

GEIA  Government  Electronics  and  Information  Technology  Association 

IBR  Integrated  Baseline  Review 

IMP  Integrated  Master  Plan 

IMS  Integrated  Master  Schedule 

NASA  National  Aeronautics  and  Space  Administration 

NCE  NGA  Campus  East 

NDIA  National  Defense  Industrial  Association 

NGA  National  Geospatial-Intelligence  Agency 

OBS  Organizational  Breakdown  Structure 

PE  Program  Event 

PMB  Performance  Measurement  Baseline 

WBS  Work  Breakdown  Structure 


427 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


REIMAGINING  WORKLOAD 
TASK  ANALYSIS: 

APPLICATIONS  TO 
TRAINING  SYSTEM  DESIGN 


\*  Dennis  Duke,  Dana  E.  Sims,  and  James  Pharmer 

Today’s  warfighter  performs  more  complex,  cognitively 
demanding  tasks  than  ever  before.  Despite  the  need  for  more 
extensive  training  to  perform  these  tasks,  acquisition  profes¬ 
sionals  are  often  tasked  to  reduce  training  budgets  and  identify 
optimal  tradeoffs.  Tools  are  available  to  help  them  make  these 
decisions  that  provide  empirical  evidence  of  how  performance 
and  mission  requirements  will  be  affected  by  design  decisions. 

This  article  offers  insights  into  the  utility  of  implementing  a 
Workload  Task  Analysis  (WLTA)  early  in  weapon  systems 
acquisition  for  the  purpose  of  focusing  on  training  system 
decisions,  and  provides  a  description  of  where  WLTA  occurs 
within  the  top-down  functional  analysis  process.  It  concludes 
with  several  examples  of  how  the  WLTA  results  can  be  used 
to  guide  training  development. 


Keywords:  Workload  Task  Analysis  (WLTA),  Training 
Development,  Top-Down  Function  Analysis,  Mission 
Requirements,  Performance  Requirements 
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A  perfect  storm  has  arisen  in  military  training  system  acquisi¬ 
tion.  First,  a  great  deal  of  attention  is  being  given  to  cost  overruns 
in  major  weapon  systems  acquisition  by  the  Department  of  Defense 
(DoD).  As  a  result,  government  officials  are  continually  searching  for 
ways  to  reduce  budgets.  Program  managers  are  continually  being 
asked  to  consider  reducing  training  time  to  reduce  costs,  and  while 
battlefield  technology  is  becoming  more  sophisticated,  it  provides 
much  greater  capability  in  considerably  less  time.  However,  in  many 
instances  the  technology  is  placing  greater  demands  on  warfighters 
by  requiring  a  shift  in  how  work  is  accomplished.  In  many  cases, 
tasks  are  more  sophisticated  and  time-sensitive  than  in  the  past. 
This  shift  in  the  type  of  work  and  speed  of  performance  by  the  war¬ 
fighter  on  the  battlefield  requires  more  training  in  knowledge-based, 
decision-making  tasks  than  ever  before.  This  type  of  training  is  more 
complex  because  it  requires  individuals  to  understand,  integrate, 
and  act  swiftly  on  the  information  generated  from  weapon  systems 
technology;  thus,  additional  time  to  train  warfighters  for  battle  is 
needed  although  budgets  seldom  allow  for  it. 

One  solution  to  this  perfect  storm  is  to  employ  systems  engi¬ 
neering  much  earlier  in  the  acquisition  process  as  advocated  by 
Michael  J.  Sullivan,  Director,  Acquisition  and  Sourcing  Management 
in  testimony  before  the  Panel  on  Defense  Acquisition  Reform,  Com¬ 
mittee  on  Armed  Services,  House  of  Representatives  (Government 
Accountability  Office  [GAO],  2009).  Numerous  dimensions  are 
inherent  to  systems  engineering  in  the  acquisition  process,  one  of 
which  involves  a  systematic  evaluation  of  the  type  of  work  done 
by  a  human  operator  on  a  new  weapon  system  with  the  intent  to 
determine  how  best  to  design  training  solutions  to  support  that 
work.  This  process  is  called  a  Workload  Task  Analysis  (WLTA)  and 
is  incorporated  as  an  element  of  an  overarching  process  called  Top- 
Down  Function  Analysis  (TDFA).  This  methodology  aligns  with  the 
revised  DoD  Instruction  5000.02  (DoD,  2008,  Enel.  8),  which  stipu¬ 
lates  that  "...  where  practicable  and  cost  effective,  system  designs 
shall  minimize  or  eliminate  system  characteristics  that  require  exces¬ 
sive  cognitive,  physical,  or  sensory  skills;  entail  extensive  training  or 
workload-intensive  tasks;  result  in  mission-critical  errors;  or  produce 
safety  or  health  hazards”  (DoD,  2008).  This  article  suggests  that 
if  this  process  is  effective  for  a  weapon  system  design,  it  can  also 
produce  similar  results  for  a  training  system. 

Although  WLTA  is  often  done  iteratively  throughout  the  weapon 
systems  acquisition  process,  this  article  is  limited  to  how  an  early 
analysis  can  initially  identify  areas  where  warfighters  experience  the 
highest  levels  of  workload  that  may  negatively  affect  performance. 
The  findings  from  conducting  WLTA  up  front  have  the  potential  to 
not  only  increase  operator  efficiency  and  effectiveness  by  influ- 
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encing  weapon  systems  design  early  in  the  acquisition  cycle,  but 
may  also  reduce  weapon  systems  life-cycle  costs.  The  design  of 
the  training  system  can  be  included  as  part  of  this  cost  reduction; 
accordingly,  this  article  provides  a  concise  definition,  examples, 
and  insight  into  WLTA  and  the  basic  steps  required  to  perform  a 
credible  WLTA. 


Fundamentals  of  Workload 
Task  Analysis  Methodology 

Modern  technology  has  vastly  changed  the  way  we  do  business 
and  has  improved  our  productivity  by  providing  us  with  many  more 
on-the-job  capabilities.  However,  the  technology  is  only  productive 
when  effectively  employed  by  the  human  operator.  This  effective 
operation  relies  upon  the  cognitive  capacity  of  individuals  com¬ 
bined  with  their  ability  to  operate  the  new  technology  to  its  fullest 
potential.  This  ability  comes  from  a  well-designed  training  program 
that  provides  the  operator  with  pertinent  information  needed  to 
effectively  perform  tasks  on  the  job.  Today,  the  amount  and  speed 
of  information  received  during  war  combined  with  the  complexity 
of  the  technology  that  military  personnel  employ  to  gather  and 
interpret  this  information  further  compounds  workload  burdens 
on  the  individual. 

Historically,  WLTA  has  been  used  to  predict  potential  perfor¬ 
mance  bottlenecks  and  pinpoint  where  to  focus  the  efforts  of 
human  factors  engineers  in  helping  them  make  informed  decisions 
on  new  systems  design.  In  numerous  instances,  operator  workload, 
task  time  demands,  and  interface  design  issues  affected  the  design 
of  numerous  platforms,  from  helicopters  to  airplanes,  ships,  and 
individual  weapon  systems  (Aldrich,  Szabo,  &  Bierbaum,  1989,  pp. 
65-80;  Laughery  &  Corker,  1997).  Recently,  trends  are  re-emerging 
in  military  settings  to  investigate  existing  systems  engineering 
processes  and  procedures  much  earlier  in  the  acquisition  program 
because  of  their  impact  on  reducing  overall  life-cycle  costs  for 
major  weapon  systems  (GAO,  2009). 

Examining  workload  can  assist  analysts  in  determining  the 
degree  to  which  operators  can  successfully  perform  their  job  to 
meet  mission  requirements  (Lysaght  et  al.,  1989).  To  design  a  good 
system,  the  designer  must  comprehend  the  concept  of  workload 
and  understand  what  “optimal”  workload  means  to  performance 
(Mitchell,  2000).  Although  a  number  of  definitions  of  workload  have 
been  provided  over  the  years,  the  overarching  theory  acknowledges 
that  it  is  a  multidimensional  construct  that  considers  the  amount  of 
effort  (e.g.,  sensory,  cognitive,  or  psychomotor)  required  or  invested 
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by  the  individual  in  order  to  perform  on  the  job  (Aldrich  et  al.,  1989; 
Nachreiner,  1995;  Wickens,  1984).  This  performance  is  affected  in 
part  by  (a)  the  demands  of  the  environment  where  the  task  is  per¬ 
formed  (e.g.,  heat,  danger),  (b)  the  complexity  of  the  task  or  the 
system  in  which  the  task  is  performed,  and  (c)  the  capability  of  the 
operator  to  satisfy  those  demands  (Parasuraman  &  Hancock,  2001; 
Parasuraman  &  Rovira,  2005;  Wickens,  2002). 

In  a  WLTA  effort,  insight  into  how  individuals  gather  and  process 
information  about  task  performance  and  the  variables  that  affect 
cognitive  decision  making  (e.g.,  environmental  factors,  task  com¬ 
plexity,  etc.)  can  be  collected  and  analyzed  through  task  network 
simulation  models.  These  models  provide  useful  human  system 
performance  information  to  human  factors  psychologists,  systems 
engineers,  and  instructional  system  developers  to  allow  for  a  myriad 
of  design  trade  decisions.  The  power  of  these  models  lies  in  their 
capability  to  predict  within  a  task  flow  where  and  when  operators 
may  not  be  able  to  perform  specific  tasks  in  a  timely,  effective,  and 
efficient  manner.  This  effectively  assists  analysts  in  determining  tar¬ 
gets  of  opportunity  for  developing  an  optimal  performance  solution. 
An  example  cited  in  a  recent  GAO  report  describes  a  situation  that 
illustrates  where  WLTA  may  affect  the  design  of  weapon  systems. 
The  report  (GAO,  2010)  describes  a  scenario  in  a  surveillance  aircraft 
where  operators  who  are  responsible  for  processing,  exploiting, 
and  disseminating  Intelligence,  Surveillance,  and  Reconnaissance 
(ISR)  data  can  only  use  collected  intelligence  data  if  the  data  are 
visible  to  them. 

Making  ISR  data  discoverable  in  this  way  can  be  accomplished 
through  meta-data  tagging. ..For  example,  a  camera  may  create 
meta-data  for  a  photograph,  such  as  date,  time,  and  lens  set¬ 
tings.  The  photographer  may  add  further  meta-data,  such  as 
the  names  of  the  subjects.  The  process  by  which  information  is 
meta-data  tagged  depends  on  the  technical  capabilities  of  the 
systems  collecting  the  information.  Most  ISR  systems  do  not 
automatically  meta-data  tag  the  ISR  data  when  they  are  trans¬ 
ferred  from  the  sensor  to  the  ground  station  for  processing  and 
exploitation  because  most  of  these  systems  were  developed 
prior  to  DoD’s  emphasis  on  enforcing  meta-data  standards. 
Since  the  sensors  on  these  legacy  systems  are  not  able  to  meta¬ 
data  tag  automatically,  it  is  up  to  each  of  the  military  services 
to  prioritize  the  cataloging  of  the  ISR  data  manually  after  col¬ 
lection.  (p.  5) 

The  solution,  influenced  by  WLTA  analysis  findings,  may  involve 
designing  software  capabilities  that  automate  these  meta-data  tags 
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and  require  the  human  operator  to  confirm  the  accuracy  of  the  tag¬ 
ging.  The  training  implications  resulting  from  this  design  change 
involve  the  identification  of  specific  knowledge  and  skills  needed 
to  operate  this  newly  designed  hardware  and  a  determination  of 
a  training  strategy  for  presenting  this  information  to  the  operator. 
However,  to  fully  understand  where  WLTA  comes  into  play  in  the 
acquisition  process,  it  is  important  to  describe  the  overarching 
analysis  process  involved.  This  is  known  as  the  TDFA. 


Understanding  the  TDFA  Process 

The  initial  assessment  of  any  workload  prediction  methodology 
requires  the  conduct  of  a  comprehensive  mission/task/workload 
analysis  (Aldrich  et  a I.,  1989).  The  TDFA  methodology  is  a  systems 
engineering  approach  that  identifies  mission  requirements  and 
provides  a  comprehensive  capability  for  ensuring  that  the  human 
performance  requirements  are  incorporated  into  the  systems  engi¬ 
neering  process  (Dugger,  Parker,  Winters,  &  Lackie,  1999).  The  intent 
of  the  TDFA  is  to  influence  and  refine  system  design  throughout 
the  acquisition  process.  The  full  TDFA  methodology  used  in  several 
naval  aviation  acquisitions  involves  nine  phases  or  steps  as  shown  in 
Figure  1  (Duke,  Guptill,  Hemenway,  &  Doddridge,  2006).  However, 
only  the  analytical  activity  undertaken  in  the  Mission  Analysis  (Phase 
1.0),  Function  Analysis  (Phase  3.0),  and  Task  Design  and  Analysis 
(Phase  5.0)  will  be  discussed  in  this  article. 


FIGURE  1.  THE  TOP-DOWN  FUNCTION  ANALYSIS  (TDFA)  PROCESS 
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In  this  TDFA  model,  the  WLTA  is  included  as  the  major  compo¬ 
nent  of  Phase  8.0,  Performance,  Workload,  and  Training  Estimation. 
However,  the  WLTA  cannot  be  undertaken  until  critical  hierarchical 
information  about  the  weapon  system’s  missions,  functions,  and 
tasks  is  available.  In  the  Phase  1.0  (Mission  Analysis),  the  external 
objectives  or  the  “what”  of  the  system  performance  are  identified. 
This  equates  to  the  systems  engineering  “requirements  analysis” 
described  in  American  National  Standards  Institute/Electronic 
Industries  Alliance  632,  Processes  for  Engineering  a  System  (ANSI/ 
EIA,  2003).  System  functions,  which  are  initially  analyzed  in  Phase 
3.0  (Function  Analysis),  describe  “how”  the  system  will  achieve 
performance  requirements.  These  system  functions  are  then  fur¬ 
ther  decomposed  into  human  and  system  tasks,  which  describe  the 
qualitative  and  quantitative  workload  of  individual,  team,  and  crew 
operators  and  maintainers.  This  decomposition  occurs  in  Phase  5.0 
(Task  Design  and  Analysis).  Optimal  design  solutions  based  upon 
recommendations  from  the  task  decomposition  are  integrated 
during  Phase  6.0  (Interface  Concepts  and  Designs)  and  Phase  7.0 
(Crew/Team  Concepts  and  Designs)  to  ensure  system-level  optimi¬ 
zation  and  compatibility.  The  results  of  the  TDFA  are  then  verified 
to  see  if  human  system  integration  is  being  adequately  addressed 
in  meeting  weapon  systems  mission  goals.  Ultimately,  the  process 
provides  a  hierarchy  for  logically  linking  human  performance  (tasks) 
with  the  combatant  commander’s  warfighting  needs  (missions)  as 
shown  in  Figure  2  (Duke  et.al.,  2006). 

FIGURE  2.  MISSIONS-FUNCTIONS-TASKS  HIERARCHY 
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The  WLTA,  which  occurs  in  Phase  8.0  (Performance,  Workload, 
and  Training  Estimation),  uses  information  obtained  in  the  previously 
mentioned  TDFA  phases.  To  gain  an  appreciation  of  the  WLTA,  a 
brief  description  of  the  mission,  function,  and  task  analysis  phases 
of  the  TDFA  model  is  provided. 

Mission  Analysis  (Phase  1.0) 

The  Mission  Analysis  Phase  of  the  TDFA  serves  to  determine 
and  document  the  overall  purpose,  objectives,  and  mission  require¬ 
ments  of  a  weapon  system.  Initially,  a  weapon  system’s  primary  and 
secondary  missions  (e.g.,  Anti-Submarine  Warfare,  Anti-Surface 
Warfare,  etc.)  are  determined  and  correlated  with  system  mis¬ 
sion  tasks.  The  Universal  Navy  Task  List  or  other  Service  task  lists 
provide  the  basis  for  identifying  the  system  mission  tasks.  Initial 
metrics  are  also  established  to  measure  results  in  the  execution  of 
missions.  This  involves  creating  Measures  of  Effectiveness  (MOEs) 
and  Measures  of  Performance  (MOPs),  which  are  used  to  determine 
the  system’s  ability  to  support  the  achievement  of  an  operational 
mission  and  the  technical  performance  standards  that  a  system 
must  achieve  to  satisfy  the  MOEs,  respectively  (Chairman  of  the 
Joint  Chiefs  of  Staff,  2003).  MOPs  also  serve  as  high-level  standards 
by  which  many  systems  operators  will  be  evaluated. 

The  system  constraints  and  boundaries,  which  may  have  an 
impact  on  training  program  design,  are  also  identified  in  the  Mis¬ 
sion  Analysis  Phase.  For  example,  in  dealing  with  the  acquisition  of 
a  training  system  for  a  new  surveillance  aircraft,  one  must  determine 
if  the  scope  of  the  platform  (system)  includes  a  ground  station.  If  it 
does,  then  the  training  analysis  must  consider  the  infrastructure  and 
all  associated  logistics  associated  with  the  ground  station  as  well  as 
that  of  the  platform.  Information  about  constraints  and  boundaries 
is  usually  obtained  by  analyzing  its  high-level  mission  objectives, 
which  are  found  in  acquisition-related  publications  such  as  the 
weapon  system’s  Initial  Capabilities  Document,  Capability  Develop¬ 
ment  Document,  Performance  Based  Specification,  and  Office  of 
the  Chief  of  Naval  Operations  Instruction  1000. 16K  (2007).  Once 
the  overall  purpose,  objectives,  and  mission  requirements  are  deter¬ 
mined,  it  must  be  determined  what  the  system  must  do  to  satisfy 
the  mission  requirements.  This  analysis  is  called  a  Function  Analysis. 

Function  Analysis  (Phase  3.0) 

The  goal  of  the  Function  Analysis  Phase  is  to  define  performance 
at  the  level  of  detail  where  it  is  possible  to  design  all  subsystems 
or  components  needed  to  satisfy  performance  requirements.  For 
example,  if  a  mission  requirement  exists  for  weapon  systems  to  per¬ 
form  surveillance,  then  the  weapon  system  (e.g.,  an  aircraft)  must  be 
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designed  with  a  means  to  undertake  the  surveillance  function  (e.g., 
must  have  a  radar  system).  Functions  provide  the  means  to  align 
mission  requirements  to  the  specific  system  hardware  and  software. 
The  Department  of  Defense  Architecture  Framework  (DODAF) 
data  model  views,  which  are  developed  by  mission  system  engi¬ 
neers,  provide  detailed  hardware  and  software  information  about 
the  functions  performed  by  equipment  embedded  in  the  technical 
systems  (e.g.,  radar  system)  comprising  a  weapon  system  (DoD, 
2009).  They  also  supply  important  information  about  the  technical 
communication  between  the  equipment  (e.g.,  describe  technical 
cues  and  responses  of  hardware  and  software  tasks).  The  integrated 
product  team  (IPT)  training  team  must  train  the  human  operator  on 
how  to  effectively  interface  with  the  equipment  and  must  ensure 
the  operator  understands  how  technical  activities  accomplish  mis¬ 
sion  requirements. 

Task  Design  and  Analysis  (Phase  5.0) 

In  the  Task  Design  and  Analysis  Phase,  analysts  develop  initial 
tasks  that  describe  how  humans  will  perform  assigned  system  func¬ 
tions.  During  this  phase,  hardware  and  software  tasks  are  linked  with 
human  performance.  Information  in  this  step  usually  comes  from  a 
stratified  sample  of  subject  matter  experts  (SMEs),  who  either  have 
performed  the  tasks  or  are  very  familiar  with  how  the  tasks  should 
be  performed.  One  way  to  collect  task  information  is  to  use  critical- 
event  scenarios  where  SMEs,  using  a  flowchart,  identify,  describe, 
and  document  the  individual  tasks  and  subtasks  they  perform  at 
their  workstation  during  a  mission.  Each  scenario  should  depict  a 
unique  mission  area,  which  allows  the  SMEs  to  collect  information 
about  the  different  types  of  workstation  tasks  performed  dur¬ 
ing  different  missions.  The  authors  recommend  that  all  technical 
publications  and  applicable  reference  workstation  documents  are 
made  available  to  SMEs  during  this  exercise.  In  cases  where  the 
SMEs  are  providing  information  based  on  their  experiences  with 
a  legacy  system,  the  legacy  tasks  are  compared  to  the  high-level 
notional  functions,  missions,  and  tasks  obtained  from  the  DoDAF 
model  views  and  documented  in  the  TDFA.  As  the  new  systems  on 
the  weapon  system  are  developed,  changes  will  be  assessed  and 
the  functional  architecture  will  be  modified. 


Undertaking  a  Workload  Task  Analysis 

As  stated  previously,  a  WLTA  effort  provides  insight  into  an 
operator’s  perceived  level  of  effort  to  complete  a  task  and  the 
variables  that  affect  decision  making.  The  WLTA  is  conducted  in 
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Phase  8.0  of  the  TDFA  process.  The  information  from  the  Mission, 
Function,  and  Task  Design  and  Analysis  Phases  enables  the  WLTA 
analyst  to  understand  the  workload  associated  with  a  given  task/ 
mission.  The  workload  activity  is  divided  into  time  and  information 
processing  demands  (e.g.,  visual,  auditory,  cognitive,  and  psycho¬ 
motor  [VACP])  so  it  can  be  examined  from  various  perspectives. 
These  perspectives  assist  the  human  factors  psychologists,  systems 
engineers,  and  instructional  system  developers  to  better  analyze 
high-workload-demand  tasks.  The  discussion  that  follows  contains 
a  brief  description  of  the  two  types  of  workload  data  collected  (i.e., 
time  estimation,  information  processing  estimation),  and  provides 
examples  of  potential  training-related  performance  solutions  to 
workload.  These  training  solutions  are  not  intended  to  be  all-inclu¬ 
sive,  but  rather  provide  a  starting-off  point  for  any  IPT  training  team 
to  consider  for  its  own  platform. 

Time  Estimation 

In  this  portion  of  the  WLTA,  the  analyst  is  interested  in  the  time 
spent  on  particular  components  of  the  overall  mission.  The  results 
from  this  analysis  can  be  an  indication  of  the  complexity  of  the 
task  or  performance  inefficiencies  (e.g.,  poor  system  design,  lack 
of  training,  etc.).  Depending  on  the  analyst’s  interpretation  of  the 
cause  of  the  time  spent  on  certain  components  of  an  overall  mission, 
targeted  training-related  solutions  may  be  identified. 

Several  options  are  available  to  gather  time-estimate  data.  Ide¬ 
ally,  the  collection  of  time  data  should  come  from  direct  observation 
of  actual  operators  performing  the  tasks  during  a  live  mission  or  on 
a  simulator  during  a  training  exercise.  In  many  cases,  however,  this  is 
not  possible,  especially  if  the  weapon  system  has  not  been  built  or 
the  nature  of  the  tasks  do  not  allow  for  direct  observation  during  a 
mission.  As  an  alternative  to  observation,  domain  SMEs  provide  esti¬ 
mates  of  the  amount  of  time  spent  performing  each  task,  as  well  as 
whether  each  task  is  continuous  (no  observable  start  or  end  point) 
or  discrete  (actions  with  definite,  observable  start-end  points).  Prior 
to  the  SMEs  providing  these  ratings,  they  are  provided  examples 
of  discrete  (e.g.,  manipulating  a  knob  on  a  computer  console) 
and  continuous  (e.g.,  monitoring  targets  on  a  radar  screen)  tasks. 
These  tasks  are  graphically  illustrated  on  flowcharts  that  clearly 
depict  what  is  actually  being  done  by  an  operator  in  response  to  a 
specific  cue.  The  flowcharts  make  it  easier  to  estimate  the  time  it 
takes  to  accomplish  a  specific  task,  and  provide  analysts  with  both 
an  average  time  on  each  discrete  task  and  a  range  of  time  for  the 
continuous  tasks. 
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Information  Processing  Estimation 

The  next  step  of  the  WLTA  is  to  determine  and  categorize  the 
amount  of  workload  required  to  perform  the  task  during  a  typi¬ 
cal  mission.  This  step  is  likely  the  most  challenging  as  it  requires 
the  analyst  to  implement  a  modeling  approach  that  accounts  for 
tasks  occurring  simultaneously,  the  types  of  tasks  being  done,  and 
other  factors  that  may  shape  the  performance  of  the  operator. 
Research  suggests  that  humans  are  able  to  process  information 
across  multiple  different  VACP  channels,  as  illustrated  in  Christo¬ 
pher  D.  Wickens’  Multiple  Resource  Theory  model  (Wickens,  2002; 
Wickens,  Sandry,  &  Vidulich,  1983).  In  such  cases,  the  summative 
workload  demands  of  multiple,  simultaneous  tasks  on  one  channel 
can  provide  some  indication  of  the  likelihood  an  individual  would 
be  able  to  perform  two  or  more  tasks  at  the  same  time  with  a  given 
workload.  For  example,  if  a  task  calls  for  an  operator  to  simultane¬ 
ously  aim  a  weapon  at  a  target  (rated  a  4  on  the  visual  channel  by 
the  operator)  and  make  some  fine  discrimination  of  symbols  on  a 
screen  (rated  a  5  on  the  visual  channel),  the  combined  demand  on 
the  operator’s  visual  channel  would  exceed  the  highest  rating  pos¬ 
sible  on  the  visual  scale  (see  Appendix).  This  high-workload  rating 
on  the  visual  channel  would  be  cause  for  concern  in  a  design  that 
required  an  operator  to  simultaneously  perform  these  two  tasks. 
The  VACP  scales  are  provided  in  the  Appendix. 


FIGURE  3.  RESULTS  OF  A  VCAP  ANALYSIS  DONE  TO  • - Visual 
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From  a  training  analyst  perspective,  it  is  necessary  to  assess  the 
workload  demands  on  an  operator  at  different  intervals  throughout 
the  mission.  To  do  so,  the  VACP  components  of  the  tasks  have  to 
be  estimated  (McCracken  &  Aldrich,  1984;  Szabo  &  Bierbaum,  1986) 
and  populated  into  a  discrete-event  simulation/modeling  software 
tool.  A  number  of  commercial  and  government  discrete-event 
simulation/modeling  software  tools  (e.g.,  MicroSaint,  Improved 
Performance  Research  Integration  Tool  [IMPRINT]),  are  available  to 
provide  the  capability  to  account  for  operator  ability  to  multitask 
across  noncompeting  processing  channels  consistent  with  Wickens’ 
Multiple  Resource  Theory  (Wickens,  2002;  U.S.  Army  Research 
Laboratory,  2005).  Figure  3  provides  an  example  of  a  simulation 
showing  how  operators  reacted  to  a  surveillance  mission  situa¬ 
tion.  It  uses  the  dynamic  modeling  technique  of  Coloured  PetriNet 
(Kristensen,  Christensen,  &  Jensen,  1998)  to  illustrate  the  predicted 
VACP  demands  on  an  electronic  intelligence  (ELINT)  operator  in  a 
surveillance  aircraft  while  performing  assigned  tasks  in  a  specified 
period  of  time  during  an  operational  mission.  Careful  evaluation  of 
these  workload  predictions  provided  the  analyst  with  insight  on 
candidate  tasks  where  workload  demands  may  be  improved.  For 
example  in  Figure  3,  the  data  indicate  that  about  7  minutes  into  the 
mission,  the  operator  had  to  undertake  several  tasks  while  respond¬ 
ing  to  target  cues  at  the  workstation.  During  30-second  intervals, 
the  operator  had  to  simultaneously  employ  several  skills,  which 
caused  a  temporary  visual  and  cognitive  overload.  The  operator 
used  high  visual  skills  (shown  by  blue  line  showing  8.5— exceeding 
the  visual  workload  scale).  Thus,  the  operator  may  not  have  been 
able  to  “see”  all  the  target  data  available  on  the  screen  during  that 
30-second  interval.  The  operator  also  used  high  cognitive  skills 
to  interpret  what  was  being  seen  and  heard  (shown  by  green  line 
showing  8.3— exceeding  the  cognitive  workload  scale).  Thus,  the 
operator  may  not  have  been  able  to  comprehend  the  information 
presented.  Both  of  these  were  done  while  interpreting  sound  pat¬ 
terns  (shown  by  red  line  indicating  7.0  on  the  auditory  scale)  and 
manually  adjusting  a  thumbwheel  (shown  by  the  light  blue  line  indi¬ 
cating  5.8  on  the  psychomotor  scale).  This  “task-stacking”  situation 
resulted  in  the  operator  exceeding  visual  and  cognitive  capacity  for 
approximately  30  seconds,  meaning  critical  information  may  have 
been  missed,  which  could  impact  overall  mission  performance.  With 
this  information,  the  engineers,  human  factors  psychologists,  and 
instructional  system  designers  can  begin  to  develop  alternatives  for 
task  redesign,  human  engineering  improvements,  and/or  training 
solutions.  In  the  following  section,  a  few  examples  of  this  process 
are  provided. 
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Workload  Solution  Identification 

The  risk  in  developing  weapon  systems  without  significant 
consideration  for  how  the  operator  will  actually  utilize  them  is  that 
the  system  utilization  may  fall  significantly  below  its  potential. 
Additionally,  a  great  deal  of  time  and  resources  may  be  required  in 
developing  training  systems  for  the  operator.  Utilizing  WLTA  data 
up  front  may  prevent  this  situation.  Workload  solutions  can  take 
many  forms,  and  should  be  based  on  the  cost,  schedule,  and  per¬ 
formance  considerations  by  the  weapon  systems  program.  The  final 
solution(s)  chosen  should  be  guided,  at  least  in  part,  by  the  results 
from  the  WLTA.  For  purposes  of  this  article,  only  training  solutions 
are  discussed,  but  WLTA  identifies  where  workload  issues  may  arise 
within  a  mission  scenario  for  operators  of  a  system  and  narrows  the 
focus  to  particular  tasks  and  combinations  of  tasks. 

WLTA  provides  information  about  the  operator  tasks  and  pro¬ 
vides  the  training  developer  with  data  about  how  user  interfaces 
are  structured  to  enable  performers  to  effectively  use  the  weapon 
system.  Specifically,  WLTA  uncovers  cues  that  initiate  task  behav¬ 
iors,  the  time  required  to  perform  the  tasks,  and  documents  various 
demands  the  tasks  place  on  the  individual.  Engineers  in  the  design 
of  weapon  systems  can  use  this  information  in  the  design  of  the 
weapon  system.  For  example,  in  the  ELINT  operator  example,  a 
software  modification  can  allow  the  workstation  to  “automatically 
identify”  targets,  thus  relieving  the  operator  from  the  require¬ 
ment  to  visually  identify  the  target.  This  information  helps  the 
training  analyst  establish  a  training  strategy  to  support  successful 
accomplishment  of  the  task.  For  this  reason,  two  training-focused 
performance  solutions  that  can  be  identified  and  implemented 
based  on  the  WLTA  data  are  what  to  train  and  how  to  train  specific 
tasks.  A  third  training  solution  that  can  be  derived  from  WLTA  data 
is  error  reduction.  Each  potential  solution  will  be  discussed  in  turn. 

What  to  Train 

The  benefit  of  the  information  provided  by  the  WLTA  is  that  it 
tells  training  analysts  where  operators  may  spend  most  of  their  time 
and  what  tasks  require  the  most  of  the  operator’s  limited  resources 
during  a  mission.  The  analysts  can  then  dig  deeper  into  this  informa¬ 
tion  to  understand  whether  these  are  areas  of  training  importance 
and  then  focus  training  on  that  area  of  tasks.  In  Figure  4,  a  deci¬ 
sion-making  matrix  that  could  be  used  by  instructional  systems 
developers  to  focus  training  is  provided.  In  this  contrived  example, 
reasonable  tradeoffs  of  what  to  train  can  be  made  to  focus  on  tasks 
that  require  High  Information  Processing  (I  and  III)  because  opera¬ 
tors  are  likely  to  require  the  most  support  in  performing  the  tasks. 
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Tasks  requiring  a  High  Time  Spent  but  Low  Information  Processing 
(II)  can  then  be  assessed  to  determine  whether  these  are  tasks  that 
could  be  automated  or  distributed  to  other  team  members  to  allow 
more  time  for  operators  to  perform  I  and  III  tasks.  Finally,  tasks 
within  the  IV  quadrant  may  be  identified  as  unnecessary  to  train, 
thereby  assisting  the  IPT  in  allocating  resources  for  the  best  return 
on  investment.  It  should  be  noted  that  this  decision-making  matrix 
is  intentionally  simplistic  for  the  purposes  of  this  article.  Factors 
such  as  the  criticality  of  the  tasks,  number  of  information  process¬ 
ing  channels  required,  and  others  important  to  the  weapon  system 
should  also  be  considered  in  determining  what  to  train.  In  the  ELINT 
operator  example,  an  early  decision  to  fund  a  software  modifica¬ 
tion  to  “automatically  identify”  targets  can  provide  life-cycle  cost 
reductions  from  a  training  perspective.  Since  targets  will  be  “auto¬ 
matically  identified”  by  the  workstations,  then  training  objectives 
relating  to  interpreting  the  information  will  be  incorporated  in  the 
curriculum.  Without  the  need  to  teach  how  to  manually  recognize 
the  targets,  the  course  can  be  shortened,  reducing  the  overall  life- 
cycle  training  costs. 


FIGURE  4.  DECISION-MAKING  MATRIX  TO  GUIDE  TRAINING 
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How  to  Train 

Once  decisions  are  made  regarding  what  to  train,  instructional 
systems  specialists  can  also  utilize  WLTA  results  to  determine  how 
best  to  train  the  skills  that  are  identified  as  training  tasks.  For  tasks 
that  fall  within  the  II  and  IV  quadrants  of  Figure  4,  low  information 
processing  is  required,  suggesting  that  these  tasks  are  relatively 
automatic  or  simplistic.  This  suggests  that  these  types  of  skills  may 
be  most  effectively  taught  partly  through  training  methods  such 
as  computer-based  online  courses  and  partly  through  task  training 
devices  or  training  simulators,  wherein  trainees  receive  a  demon¬ 
stration  of  how/when  to  perform  the  tasks  and  opportunities  to 
practice  performing  the  skills.  Conversely,  demanding  tasks  (i.e., 
high  information  processing  tasks  [I  and  III])  often  involve  more  cue 
complexity  and  mental  effort  (Wickens  &  Carswell,  2006).  With  this 
information  in  hand,  the  instructional  developer  can  appropriate 
more  time  to  train  complex  tasks  and  ensure  prerequisite  knowl¬ 
edge  and  skills  are  acquired  early  in  training.  Furthermore,  training 
strategies  can  be  chosen  to  ensure  more  trainee-instructor  interac¬ 
tion,  and  may  allow  operators  repeated  opportunities  to  practice 
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the  tasks  with  varying  and  increasing  levels  of  complexity  to  build 
the  decision-making  and  information  processing  skills  that  are  less 
outwardly  tangible  and  difficult  to  train. 

Reducing  Errors 

Interviews  with  SMEs  during  the  WLTA  often  reveal  common 
mistakes  (as  well  as  their  consequences)  made  by  operators.  Senior 
operators  will  comment  that  mistakes  in  performance  are  usually 
traceable  to  inattention,  over  attention,  or  fixation  (Greenwell, 
Strunk,  &  Knight,  2004;  Wickens  &  Carswell,  2006).  As  Carl  (2009, 
p.  120)  noted,  “...there  is  a  tendency  for  performers  to  devote  too 
much  time  to  some  cues,  devote  too  little  time  to  other  cues,  or 
poorly  manage  their  time  in  attending  to  all  the  cues  that  impact 
task  execution.”  With  this  information,  instructional  designers  can 
focus  on  initially  training  new  operators  to  select  and  concentrate 
on  important  task  cues  while  disregarding  irrelevant  noise.  In  this 
case,  utilizing  WLTA  data  will  not  only  emphasize  the  important 
components  of  training,  it  will  reduce  downstream  performance 
problems  that  will  increase  life-cycle  costs  for  repair  or  replace¬ 
ment  of  the  system.  In  the  ELINT  operator  example,  the  costs  for  a 
software  modification  to  “automate  target  recognition”  are  incurred 
only  once  during  acquisition  (unless  there  are  system  upgrades  in 
which  additional  costs  will  be  incurred).  Training  costs  associated 
with  teaching  target  recognition  will  reoccur  with  each  new  set  of 
trained  operators,  thus  affecting  overall  life-cycle  training  costs. 


Conclusions 

Although  acquisition  professionals  are  continually  asked  to 
identify  tradeoffs  to  reduce  weapon  system  budgets,  tools  are 
available  to  help  them  make  decisions  regarding  life-cycle  costs. 
Everyone  acknowledges  that  weapon  systems  being  acquired  today 
are  extremely  sophisticated.  The  operator  and  maintainer  tasks 
associated  with  the  weapon  systems  are  also  becoming  increasingly 
complex,  and  they  require  time  and  expensive  simulators  to  satisfy 
training  requirements.  Training  time  and  training  media  are  quite 
costly  from  a  life-cycle  perspective. 

WLTA  could  be  an  extremely  valuable  tool  in  reducing  life-cycle 
costs,  ensuring  the  system  can  be  used  effectively  by  designing 
training  that  can  support  the  operator.  Admittedly,  WLTA  is  no  sim¬ 
ple  task,  requires  significant  time  and  support  from  the  acquisition 
team,  and  must  be  adapted  to  the  needs  of  the  individual  program. 
However,  the  results  of  WLTAs  can  become  increasingly  valuable  as 
the  team  is  required  to  make  trade-off  decisions  and  must  negotiate 
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with  program  managers  to  retain  funding,  which  can  be  done  with 
the  support  of  systematically  derived  evidence  of  how  performance 
and  mission  requirements  will  be  affected  by  design  decisions. 
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APPENDIX 


Visual  Workload  Scale 

Scale  Value 

Visual  Scale  Descriptor 

0.0 

No  Visual  Activity 

1.0 

Visually  Register/Detect  (detect  occurrence  of  image) 

3.7 

Visually  Discriminate  (detect  visual  differences) 

4.0 

Visually  Inspect/Check  (discrete  inspection/static 
condition) 

5.0 

Visually  Locate/Align  (selective  orientation) 

5.4 

Visually  Track/Follow  (maintain  orientation) 

5.9 

Visually  Read  (symbol) 

7.0 

Visually  Scan/Search/Monitor  (continuous/serial 
inspection,  multiple  conditions) 

Auditory  Workload  Scale 


Scale  Value 

Auditory  Scale  Descriptor 

0.0 

No  Auditory  Activity 

1.0 

Detect/Register  Sound  (detect  occurrence  of  sound) 

2.0 

Orient  to  Sound  (general  orientation/attention) 

4.2 

Orient  to  Sound  (selective  orientation/attention) 

4.3 

Verify  Auditory  Feedback  (detect  occurrence  of 
anticipated  sound) 

4.9 

Interpret  Semantic  Content  (speech) 

6.6 

Discriminate  Sound  Characteristics  (detect  auditory 
differences) 

7.0 

Interpret  Sound  Patterns  (pulse  rates,  etc.) 
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Cognitive  Workload  Scale 

Scale  Value 

Cognitive  Scale  Descriptor 

0.0 

No  Cognitive  Activity 

1.0 

Automatic  (simple  association) 

1.2 

Alternative  Selection 

3.7 

Sign/Signal  Recognition 

4.6 

Evaluation/Judgment  (consider  single  aspect) 

5.3 

Encoding/Decoding,  Recall 

6.8 

Evaluation/Judgment  (consider  several  aspects) 

7.0 

Estimation,  Calculation,  Conversion 

Psychomotor  Workload  Scale 

Scale  Value 

Psychomotor  Scale  Descriptor 

0.0 

No  Psychomotor  Activity 

1.0 

Speech 

2.2 

Discrete  Actuation  (button,  toggle,  trigger) 

2.6 

Continuous  Adjustive  (Flight  control,  sensor  control) 

4.6 

Manipulative 

5.8 

Discrete  Adjustive  (rotary,  vertical  thumbwheel,  lever 
position) 

6.5 

Symbolic  Production  (writing) 

7.0 

Serial  Discrete  Manipulation  (keyboard  entries) 
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of  the  Defense  Acquisition  Workforce.  Each  issue  of  the  Defense 
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believe  should  be  required  reading  for  the  defense  acquisition 
professional.  The  reviews  should  be  400  words  or  fewer,  describe 
the  book  and  its  major  ideas,  and  explain  its  relevance  to  defense 
acquisition.  Please  send  your  reviews  to  the  Managing  Editor, 
Defense  Acquisition  Research  Journal'. 
Norene.Fagan-Blanch@dau.mil. 
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Review: 

With  an  attention  to  detail  seldom  encountered,  coupled 
with  penetrating  psychological  explorations  into  the  minds  and 
motives  of  many  of  those  involved,  Pulitzer  prize  winning  author 
Neil  Sheehan  provides  a  comprehensive  look  at  the  Cold  War 
development  of  the  Intercontinental  Ballistic  Missile  (ICBM),  written 
around  the  story  of  the  life  and  career  of  Gen.  Bernard  Schriever, 
commander  of  the  Air  Force  Systems  Command,  the  brilliant  man 
who  brought  the  ICBM  to  life.  He  does  this  while  exploring  the  birth 
of  the  United  States  Air  Force  and  the  formation  of  the  Strategic 
Air  Command.  The  importance  of  the  ICBM  among  U.S.  weapon 
systems,  and  how  the  people  involved  came  together  to  give  it  birth, 
is  masterfully  recounted. 

Schriever’s  influence  was  palpable.  He  had  battled  the  likes  of 
Curtis  LeMay,  first  commander  of  Strategic  Air  Command,  who 
believed  bombers  were  the  ultimate  strategic  weapon.  Sheehan 
shows  how  they  lacked  the  vision  to  see  how  useless  bombers 
would  be  in  the  event  of  a  strategic  nuclear  war  where  the  ICBM, 
capable  of  striking  a  target  continents  away  in  a  matter  of  minutes, 
would  be  the  primary— and  deciding— weapon.  By  1963,  Schriever 
controlled  40  percent  of  the  Air  Force  budget. 

Sheehan  captures  in  fascinating  detail  the  relationship  between 
Schriever  and  the  head  of  the  U.S.  Army  Air  Force,  Henry  “Hap” 
Arnold,  and  shows  with  clarity  seldom  seen  elsewhere  the  influence 
a  visionary  leader  like  Arnold  is  able  to  exert  to  shape  the  career 
and  open  the  doors  to  advancement  of  someone  as  brilliant  and 
visionary  as  Schreiver.  He  shows  how  Schriever’s  vision  and  strategic 
thinking  ability  enabled  him  to  see  with  absolute  clarity  the  need  to 
develop  the  ICBM  to  protect  his  adopted  country  from  the  growing 
menace  of  the  Soviet  Union,  in  spite  of  encountering  resistance 
from  LeMay  at  every  turn.  Sheehan  also  describes  how  Schriever 
set  up  research  and  development  labs  as  a  critical  element  in  the 
advancement  of  weapon  systems,  while  addressing  the  problems 
with  Soviet  spies  who  had  infiltrated  the  research  labs.  He  was  able 
to  stay  on  target,  to  continually  shift  tactics  to  reach  his  strategic 
goal,  working  through  and  around  challenges  from  people,  budgets, 
family  obligations,  and  Air  Force  top  brass. 

All  those  who  worked  with  Schriever  really  did  walk  with  a 
legend  whose  story  deserved  to  be  memorialized.  For  this,  we  owe 
Sheehan  a  debt  of  gratitude. 
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in  one  or  more  of  the  areas  of  acquisition  discussed  above.  Research  articles 
should  not  exceed  4,500  words. 

Audience  and  Writing  Style 

The  readers  of  the  Defense  ARJ  are  primarily  practitioners  within  the 
defense  acquisition  community.  Authors  should  therefore  strive  to  demon¬ 
strate,  clearly  and  concisely,  how  their  work  affects  this  community.  At  the 
same  time,  do  not  take  an  overly  scholarly  approach  in  either  content  or 
language. 
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Format 

Please  submit  your  manuscript  with  references  in  APA  format  (author- 
date-page  number  form  of  citation)  as  outlined  in  the  Publication  Manual  of  the 
American  Psychological  Association  (6th  Edition).  For  all  other  style  questions, 
please  refer  to  the  Chicago  Manual  of  Style  (15th  Edition). 

Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librarian 
in  completing  citation  of  government  documents  because  standard  formulas 
of  citations  may  provide  incomplete  information  in  reference  to  government 
works.  Helpful  guidance  is  also  available  in  Garner,  D.  L.,  and  Smith,  D.  H.,  1993, 
The  Complete  Guide  to  Citing  Government  Documents:  A  Manual  for  Writers 
and  Librarians  (Rev.  Ed.),  Bethesda,  MD:  Congressional  Information  Service. 

Pages  should  be  double-spaced  and  organized  in  the  following  order:  title 
page,  abstract  (120  words  or  less),  two-line  summary,  list  of  keywords  (five 
words  or  less),  body  of  the  paper,  reference  list  (works  cited),  author’s  note  (if 
any),  and  any  figures  or  tables. 

Figures  or  tables  should  not  be  inserted  (or  embedded,  etc.)  into  the  text, 
but  segregated  (one  to  a  page)  following  the  text.  When  material  is  submitted 
electronically,  each  figure  or  table  should  be  saved  to  a  separate,  exportable 
file  (i.e.,  a  readable  EPS  file).  For  additional  information  on  the  preparation  of 
figures  or  tables,  see  CBE  Scientific  Illustration  Committee,  1988,  Illustrating 
Science:  Standards  for  Publication,  Bethesda,  MD:  Council  of  Biology  Editors. 
Restructure  briefing  charts  and  slides  to  look  similar  to  those  in  previous  issues 
of  the  Defense  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authors)  should 
attach  to  the  manuscript  a  signed  cover  letter  that  provides  all  of  the  authors’ 
names,  mailing  and  e-mail  addresses,  as  well  as  telephone  and  fax  numbers. 
The  letter  should  verify  that  the  submission  is  an  original  product  of  the  author; 
that  it  has  not  been  previously  published  in  another  journal  (monographs  and 
conference  proceedings,  however,  are  okay);  and  that  it  is  not  under  consider¬ 
ation  by  another  journal  for  publication.  Details  about  the  manuscript  should 
also  be  included  in  this  letter:  for  example,  title,  word  length,  a  description  of 
the  computer  application  programs,  and  file  names  used  on  enclosed  CDs, 
e-mail  attachments,  or  other  electronic  media. 

COPYRIGHT 

The  Defense  ARJ  is  a  publication  of  the  United  States  Government  and 
as  such  is  not  copyrighted.  Because  the  Defense  ARJ  is  posted  as  a  complete 
document  on  the  DAU  homepage,  we  will  not  accept  copyrighted  manuscripts 
that  require  special  posting  requirements  or  restrictions.  If  we  do  publish  your 
copyrighted  article,  we  will  print  only  the  usual  caveats.  The  work  of  federal 
employees  undertaken  as  part  of  their  official  duties  is  not  subject  to  copyright 
except  in  rare  cases. 

Web-only  publications  will  be  held  to  the  same  high  standards  and  scrutiny 
as  articles  that  appear  in  the  printed  version  of  the  journal  and  will  be  posted 
to  the  DAU  website  at  www.dau.mil. 
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In  citing  the  work  of  others,  please  be  precise  when  following  the  author- 
date-page  number  format.  It  is  the  contributor’s  responsibility  to  obtain 
permission  from  a  copyright  holder  if  the  proposed  use  exceeds  the  fair  use 
provisions  of  the  law  (see  U.S.  Government  Printing  Office,  1994,  Circular  92: 
Copyright  Law  of  the  United  States  of  America,  p.  15,  Washington,  D.C.).  Con¬ 
tributors  will  be  required  to  submit  a  copy  of  the  writer's  permission  to  the 
Managing  Editor  before  publication. 

Policy 

We  reserve  the  right  to  decline  any  article  that  fails  to  meet  the  following 
copyright  requirements: 

•  The  author  cannot  obtain  permission  to  use  previously  copyrighted 
material  (e.g.,  graphs  or  illustrations)  in  the  article. 

•  The  author  will  not  allow  DAU  to  post  the  article  in  our  Defense  ARJ  issue 
on  our  Internet  homepage. 

•  The  author  requires  that  usual  copyright  notices  be  posted  with  the 
article. 

•  To  publish  the  article  requires  copyright  payment  by  the  DAU  Press. 

SUBMISSION 

All  manuscript  submissions  should  include  the  following: 

•  Cover  letter 

•  Biographical  sketch  for  each  author 

•  Headshot  for  each  author  should  be  saved  to  a  CDR  disk  as  a  300  dpi 
(dots  per  inch)  or  high-print  quality  JPEG  or  Tiff  file  saved  as  no  less 
than  5x7.  Please  note:  images  from  Web,  PowerPoint,  or  e-mail  will  not 
be  accepted  due  to  low  image  quality. 

•  One  copy  of  the  typed  manuscript,  including: 

°  Abstract  of  article 

°  Two-line  summary 
°  Keywords  (5  words  or  less) 

These  items  should  be  sent  electronically,  as  appropriately  labeled  files, 
to  Defense  ARJ  Managing  Editor,  Norene  Fagan-Blanch  at:  Norene.Fagan- 
Blanch@dau.mil. 
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